curl-loader-devel Mailing List for curl-loader - web application testing (Page 6)
Status: Alpha
Brought to you by:
coroberti
Menu
▾
▴
curl-loader-devel — curl-loader usage and development issues
You can subscribe to this list here.
| 2006 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
(1) |
Dec
|
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2007 |
Jan
|
Feb
(1) |
Mar
(7) |
Apr
(19) |
May
(25) |
Jun
(16) |
Jul
(59) |
Aug
(29) |
Sep
(18) |
Oct
(19) |
Nov
(7) |
Dec
(29) |
| 2008 |
Jan
(6) |
Feb
(18) |
Mar
(8) |
Apr
(27) |
May
(26) |
Jun
(5) |
Jul
(6) |
Aug
|
Sep
(9) |
Oct
(37) |
Nov
(61) |
Dec
(17) |
| 2009 |
Jan
(21) |
Feb
(25) |
Mar
(4) |
Apr
(2) |
May
(8) |
Jun
(15) |
Jul
(18) |
Aug
(23) |
Sep
(10) |
Oct
(16) |
Nov
(14) |
Dec
(22) |
| 2010 |
Jan
(23) |
Feb
(8) |
Mar
(18) |
Apr
(1) |
May
(34) |
Jun
(23) |
Jul
(11) |
Aug
(1) |
Sep
(13) |
Oct
(10) |
Nov
(2) |
Dec
(8) |
| 2011 |
Jan
|
Feb
(7) |
Mar
(24) |
Apr
(12) |
May
(3) |
Jun
(2) |
Jul
(2) |
Aug
|
Sep
(5) |
Oct
(20) |
Nov
(7) |
Dec
(11) |
| 2012 |
Jan
(12) |
Feb
(5) |
Mar
(16) |
Apr
(3) |
May
|
Jun
(5) |
Jul
(12) |
Aug
(6) |
Sep
|
Oct
|
Nov
(8) |
Dec
|
| 2013 |
Jan
(1) |
Feb
(3) |
Mar
(5) |
Apr
(3) |
May
(1) |
Jun
|
Jul
(1) |
Aug
(2) |
Sep
(9) |
Oct
|
Nov
(8) |
Dec
(4) |
| 2014 |
Jan
(4) |
Feb
|
Mar
(1) |
Apr
|
May
(1) |
Jun
|
Jul
|
Aug
|
Sep
(4) |
Oct
|
Nov
(11) |
Dec
(5) |
| 2015 |
Jan
(1) |
Feb
|
Mar
(11) |
Apr
(3) |
May
(1) |
Jun
(1) |
Jul
(4) |
Aug
(1) |
Sep
(7) |
Oct
(4) |
Nov
(2) |
Dec
|
| 2016 |
Jan
|
Feb
(1) |
Mar
|
Apr
|
May
|
Jun
|
Jul
(1) |
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
| 2018 |
Jan
|
Feb
|
Mar
|
Apr
|
May
(1) |
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
| 2020 |
Jan
|
Feb
|
Mar
|
Apr
(1) |
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
| 2022 |
Jan
(1) |
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
|
From: Robert I. <cor...@gm...> - 2012-07-10 14:53:32
|
Hi Pankaj, It seems that the subject of your e-mail is not correct. Please, provide a meaningful subject. libcurl API indeed provides means to relax SSL requirements and not to alert users on self-signed certificates. Could you provide more details, please? When do you seen the below message On Mon, Jul 9, 2012 at 11:38 AM, pankaj sharma <pan...@gm...> wrote: > Hi Experts, > > Is there any way to ignore pad lock warning message using curl-loader while > accessing a ssl site? I mean when we configure a web site using self signed > certificate then it displays a pad lock message on to the browser like > below: > > "We recommend that you close this webpage and do not continue to this > website. > Click here to close this webpage. > Continue to this website (not recommended). > More information" > > can we ignore this message while lauching the test for a ssl site using > curl-loader? > > Thanks & Regards > Pankaj > > On Mon, Jul 9, 2012 at 1:55 PM, Neil Croft <in...@ma...> wrote: >>>> >>>> Hi, >>>> >>>> Wonder if you can assist im looking to switch mobile user-agents in >>>> curl-loader requests so that i can test a mobile piwik plugin i wrote. is >>>> this possible? >>>> im trying to call the following url >>>> http://stats.mahalamobile.co.za/piwik.php?idsite=1&rec=1 i have tried the >>>> get and post but cant seem to get piwik to find the impression? >>>> any ideas? >>>> >>>> thanks >>>> Regards, >>>> Neil Croft >>> >>> >> >> >> >> ------------------------------------------------------------------------------ >> Live Security Virtual Conference >> Exclusive live event will cover all the ways today's security and >> threat landscape has changed and how IT managers can respond. Discussions >> will include endpoint security, mobile security and the latest in malware >> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ >> _______________________________________________ >> curl-loader-devel mailing list >> cur...@li... >> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel >> > > > ------------------------------------------------------------------------------ > Live Security Virtual Conference > Exclusive live event will cover all the ways today's security and > threat landscape has changed and how IT managers can respond. Discussions > will include endpoint security, mobile security and the latest in malware > threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ > _______________________________________________ > curl-loader-devel mailing list > cur...@li... > https://lists.sourceforge.net/lists/listinfo/curl-loader-devel > -- Regards, Robert Iakobashvili, Ph.D. Home: http://www.ghotit.com Blog: http://www.ghotit.com/category/ghotit-blogs/ Twitter: http://twitter.com/ghotit Facebook: http://facebook.com/ghotit ...................................................................... Ghotit Dyslexia Assistive technology that understands you ...................................................................... |
|
From: Robert I. <cor...@gm...> - 2012-07-10 14:50:30
|
Hi Neil, Could you please clarify with more details of what you are doing since it's not clear from your posting? Thanks, Robert On Mon, Jul 9, 2012 at 11:25 AM, Neil Croft <in...@ma...> wrote: >>> Hi, >>> >>> Wonder if you can assist im looking to switch mobile user-agents in >>> curl-loader requests so that i can test a mobile piwik plugin i wrote. is >>> this possible? >>> im trying to call the following url >>> http://stats.mahalamobile.co.za/piwik.php?idsite=1&rec=1 i have tried the >>> get and post but cant seem to get piwik to find the impression? >>> any ideas? >>> >>> thanks >>> Regards, >>> Neil Croft >> >> > > > ------------------------------------------------------------------------------ > Live Security Virtual Conference > Exclusive live event will cover all the ways today's security and > threat landscape has changed and how IT managers can respond. Discussions > will include endpoint security, mobile security and the latest in malware > threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ > _______________________________________________ > curl-loader-devel mailing list > cur...@li... > https://lists.sourceforge.net/lists/listinfo/curl-loader-devel > -- Regards, Robert Iakobashvili, Ph.D. Home: http://www.ghotit.com Blog: http://www.ghotit.com/category/ghotit-blogs/ Twitter: http://twitter.com/ghotit Facebook: http://facebook.com/ghotit ...................................................................... Ghotit Dyslexia Assistive technology that understands you ...................................................................... |
|
From: pankaj s. <pan...@gm...> - 2012-07-09 08:38:24
|
Hi Experts, Is there any way to ignore pad lock warning message using curl-loader while accessing a ssl site? I mean when we configure a web site using self signed certificate then it displays a pad lock message on to the browser like below: "We recommend that you close this webpage and do not continue to this website. Click here to close this webpage. Continue to this website (not recommended). More information" can we ignore this message while lauching the test for a ssl site using curl-loader? Thanks & Regards Pankaj On Mon, Jul 9, 2012 at 1:55 PM, Neil Croft <in...@ma...> wrote: > Hi, >>> >>> Wonder if you can assist im looking to switch mobile user-agents in >>> curl-loader requests so that i can test a mobile piwik plugin i wrote. is >>> this possible? >>> im trying to call the following url >>> http://stats.mahalamobile.co.za/piwik.php?idsite=1&rec=1 i have tried >>> the get and post but cant seem to get piwik to find the impression? >>> any ideas? >>> >>> thanks >>> Regards, >>> Neil Croft >>> >> >> > > > ------------------------------------------------------------------------------ > Live Security Virtual Conference > Exclusive live event will cover all the ways today's security and > threat landscape has changed and how IT managers can respond. Discussions > will include endpoint security, mobile security and the latest in malware > threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ > _______________________________________________ > curl-loader-devel mailing list > cur...@li... > https://lists.sourceforge.net/lists/listinfo/curl-loader-devel > > |
|
From: Neil C. <in...@ma...> - 2012-07-09 08:25:27
|
> > Hi, >> >> Wonder if you can assist im looking to switch mobile user-agents in >> curl-loader requests so that i can test a mobile piwik plugin i wrote. is >> this possible? >> im trying to call the following url >> http://stats.mahalamobile.co.za/piwik.php?idsite=1&rec=1 i have tried >> the get and post but cant seem to get piwik to find the impression? >> any ideas? >> >> thanks >> Regards, >> Neil Croft >> > > |
|
From: Robert I. <cor...@gm...> - 2012-06-28 11:25:11
|
Dear Ciaran, Look at URL-probabality feature in README. Regards, Robert On Wed, Jun 27, 2012 at 8:24 PM, Ciaran McCrisken <cmc...@gm...> wrote: > Hi, > > > > Curlloader is a great piece of software – keep up the good work! > > > > I was wondering if it’s possible to fetch a list of URLS simultaneously? I > have a script that automates the generation of large amounts of URLs. I’m > looking to transfer a known amount of unique data and a known throughput via > a proxy device. > > > > At the moment, Curlloader processes each URL sequentially for each Client. > I’d like it to download each URL for each specified Client, once. > > > > E.g. > > > > ########## GENERAL SECTION ########## > > BATCH_NAME=sim_get > > CLIENTS_NUM_MAX=1000 > > CLIENTS_NUM_START=1000 > > CLIENTS_RAMPUP_INC=1 > > INTERFACE=eth0 > > NETMASK=255.255.224.0 > > IP_ADDR_MIN=172.16.47.100 > > IP_ADDR_MAX=172.16.47.100 > > CYCLES_NUM=1 > > URLS_NUM=1 > > > > ########## URLs SECTION ############# > > > > URL=http://172.16.62.110/cgi-bin/bytegenerator?size=100&seed=1 > > URL_SHORT_NAME="1" > > REQUEST_TYPE=GET > > TRANSFER_LIMIT_RATE=1000 > > > > URL=http://172.16.62.110/cgi-bin/bytegenerator?size=100&seed=2 > > URL_SHORT_NAME="2" > > REQUEST_TYPE=GET > > TRANSFER_LIMIT_RATE=1000 > > > > URL=http://172.16.62.110/cgi-bin/bytegenerator?size=100&seed=3 > > URL_SHORT_NAME="3" > > REQUEST_TYPE=GET > > TRANSFER_LIMIT_RATE=1000 > > > > .. > > .. > > .. > > > > URL=http://172.16.62.110/cgi-bin/bytegenerator?size=100&seed=1000 > > URL_SHORT_NAME="1000" > > REQUEST_TYPE=GET > > TRANSFER_LIMIT_RATE=1000 > > > > Thanks and regards! > > > ------------------------------------------------------------------------------ > Live Security Virtual Conference > Exclusive live event will cover all the ways today's security and > threat landscape has changed and how IT managers can respond. Discussions > will include endpoint security, mobile security and the latest in malware > threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ > _______________________________________________ > curl-loader-devel mailing list > cur...@li... > https://lists.sourceforge.net/lists/listinfo/curl-loader-devel > -- Regards, Robert Iakobashvili, Ph.D. Home: http://www.ghotit.com Blog: http://www.ghotit.com/category/ghotit-blogs/ Twitter: http://twitter.com/ghotit Facebook: http://facebook.com/ghotit ...................................................................... Ghotit Dyslexia Assistive technology that understands you ...................................................................... |
|
From: Ciaran M. <cmc...@gm...> - 2012-06-27 17:24:32
|
Hi, Curlloader is a great piece of software - keep up the good work! I was wondering if it's possible to fetch a list of URLS simultaneously? I have a script that automates the generation of large amounts of URLs. I'm looking to transfer a known amount of unique data and a known throughput via a proxy device. At the moment, Curlloader processes each URL sequentially for each Client. I'd like it to download each URL for each specified Client, once. E.g. ########## GENERAL SECTION ########## BATCH_NAME=sim_get CLIENTS_NUM_MAX=1000 CLIENTS_NUM_START=1000 CLIENTS_RAMPUP_INC=1 INTERFACE=eth0 NETMASK=255.255.224.0 IP_ADDR_MIN=172.16.47.100 IP_ADDR_MAX=172.16.47.100 CYCLES_NUM=1 URLS_NUM=1 ########## URLs SECTION ############# URL=http://172.16.62.110/cgi-bin/bytegenerator?size=100&seed=1 URL_SHORT_NAME="1" REQUEST_TYPE=GET TRANSFER_LIMIT_RATE=1000 URL=http://172.16.62.110/cgi-bin/bytegenerator?size=100&seed=2 URL_SHORT_NAME="2" REQUEST_TYPE=GET TRANSFER_LIMIT_RATE=1000 URL=http://172.16.62.110/cgi-bin/bytegenerator?size=100&seed=3 URL_SHORT_NAME="3" REQUEST_TYPE=GET TRANSFER_LIMIT_RATE=1000 .. .. .. URL=http://172.16.62.110/cgi-bin/bytegenerator?size=100&seed=1000 URL_SHORT_NAME="1000" REQUEST_TYPE=GET TRANSFER_LIMIT_RATE=1000 Thanks and regards! |
|
From: 邹靖晗 <zou...@gm...> - 2012-06-27 02:21:07
|
Thanks a lot for your reply. But it seems 0 is legal value for TIMER_URL_COMPLETION . As far as my test, I changed the value to 200, the core dump still occurs. Here is the discrebtion for TIMER_URL_COMPLETION from the FAQ: TIMER_URL_COMPLETION is the time in milli-seconds to allow a url fetching operation, including all internal re-directions. The legal values are 0, which means no limit on url-completion time, or above 20 (milli-seconds). When a value of the tag is above 0, we are monitoring the progress of url-fetching and canceling it, if not accompliched within the the specified value of milliseconds. The results are presented at the Load Status GUI as the operation "Timed Out" statistics and logged to all statistics files as T-Err number. 2012/6/26 divfor <di...@gm...> > TIMER_URL_COMPLETION=0 > > must be lager then zero > > Sent from mobile > > 在 2012-6-26，16:41，邹靖晗 <zou...@gm...> 写道： > > Hi all, > > I am using curl-loader to try some network test, I use perl to drive > the test. I saved the general tags and a set of url tags in a hash table, > then use foreach to loop the hash table to output the configuration file > for curl-loader. So the order of tags is based on the hash method in perl. > Then I run curl-loader -f <the configuration file> and get a core dump. > > Seems the order for the tags will impact the test result. So I want to > know what is the rule for the order of these tags in curl-loader? > > > > Here is the configuration file: > > [root@localhost jhzou]# cat 10.conf > CYCLES_NUM=-1 > INTERFACE=eth1 > CLIENTS_NUM_START=10 > URLS_NUM=1 > IP_ADDR_MIN=10.1.1.130 > CLIENTS_RAMPUP_INC=5 > BATCH_NAME=default_test > NETMASK=255.255.255.0 > IP_ADDR_MAX=10.1.1.149 > CLIENTS_NUM_MAX=20 > > > > TIMER_URL_COMPLETION=0 > URL=http://10.1.1.122/index.htm > URL_SHORT_NAME=default_url > TIMER_AFTER_URL_SLEEP=0 > REQUEST_TYPE=GET > > > This is the core dump debug informantion > > Program received signal SIGSEGV, Segmentation fault. > timer_url_completion_parser (bctx=0x7ffffffe0b00, value=<value optimized > out>) at parse_conf.c:1899 > 1899 > bctx->url_ctx_array[bctx->url_index].timer_url_completion_hrange = > (gdb) p bctx->url_index > $1 = 0 > (gdb) p bctx->url_ctx_array[bctx->url_index] > Cannot access memory at address 0x0 > (gdb) bt > #0 timer_url_completion_parser (bctx=0x7ffffffe0b00, value=<value > optimized out>) at parse_conf.c:1899 > #1 0x000000000041c45f in add_param_to_batch (filename=<value optimized > out>, bctx_array=0x7ffffffe0b00, > bctx_array_size=64) at parse_conf.c:416 > #2 parse_config_file (filename=<value optimized out>, > bctx_array=0x7ffffffe0b00, bctx_array_size=64) > at parse_conf.c:2897 > #3 0x000000000040c524 in main (argc=<value optimized out>, > argv=0x7fffffffe588) at loader.c:159 > > -- > Sincerely yours, > Zou Jinghan > > > ------------------------------------------------------------------------------ > Live Security Virtual Conference > Exclusive live event will cover all the ways today's security and > threat landscape has changed and how IT managers can respond. Discussions > will include endpoint security, mobile security and the latest in malware > threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ > > _______________________________________________ > curl-loader-devel mailing list > cur...@li... > https://lists.sourceforge.net/lists/listinfo/curl-loader-devel > > > > ------------------------------------------------------------------------------ > Live Security Virtual Conference > Exclusive live event will cover all the ways today's security and > threat landscape has changed and how IT managers can respond. Discussions > will include endpoint security, mobile security and the latest in malware > threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ > _______________________________________________ > curl-loader-devel mailing list > cur...@li... > https://lists.sourceforge.net/lists/listinfo/curl-loader-devel > > -- Sincerely yours, Zou Jinghan |
|
From: divfor <di...@gm...> - 2012-06-26 12:36:54
|
> TIMER_URL_COMPLETION=0 > must be lager then zero Sent from mobile 在 2012-6-26，16:41，邹靖晗 <zou...@gm...> 写道： > Hi all, > > I am using curl-loader to try some network test, I use perl to drive the test. I saved the general tags and a set of url tags in a hash table, then use foreach to loop the hash table to output the configuration file for curl-loader. So the order of tags is based on the hash method in perl. Then I run curl-loader -f <the configuration file> and get a core dump. > > Seems the order for the tags will impact the test result. So I want to know what is the rule for the order of these tags in curl-loader? > > > > Here is the configuration file: > > [root@localhost jhzou]# cat 10.conf > CYCLES_NUM=-1 > INTERFACE=eth1 > CLIENTS_NUM_START=10 > URLS_NUM=1 > IP_ADDR_MIN=10.1.1.130 > CLIENTS_RAMPUP_INC=5 > BATCH_NAME=default_test > NETMASK=255.255.255.0 > IP_ADDR_MAX=10.1.1.149 > CLIENTS_NUM_MAX=20 > > > > TIMER_URL_COMPLETION=0 > URL=http://10.1.1.122/index.htm > URL_SHORT_NAME=default_url > TIMER_AFTER_URL_SLEEP=0 > REQUEST_TYPE=GET > > > This is the core dump debug informantion > > Program received signal SIGSEGV, Segmentation fault. > timer_url_completion_parser (bctx=0x7ffffffe0b00, value=<value optimized out>) at parse_conf.c:1899 > 1899 bctx->url_ctx_array[bctx->url_index].timer_url_completion_hrange = > (gdb) p bctx->url_index > $1 = 0 > (gdb) p bctx->url_ctx_array[bctx->url_index] > Cannot access memory at address 0x0 > (gdb) bt > #0 timer_url_completion_parser (bctx=0x7ffffffe0b00, value=<value optimized out>) at parse_conf.c:1899 > #1 0x000000000041c45f in add_param_to_batch (filename=<value optimized out>, bctx_array=0x7ffffffe0b00, > bctx_array_size=64) at parse_conf.c:416 > #2 parse_config_file (filename=<value optimized out>, bctx_array=0x7ffffffe0b00, bctx_array_size=64) > at parse_conf.c:2897 > #3 0x000000000040c524 in main (argc=<value optimized out>, argv=0x7fffffffe588) at loader.c:159 > > -- > Sincerely yours, > Zou Jinghan > ------------------------------------------------------------------------------ > Live Security Virtual Conference > Exclusive live event will cover all the ways today's security and > threat landscape has changed and how IT managers can respond. Discussions > will include endpoint security, mobile security and the latest in malware > threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ > _______________________________________________ > curl-loader-devel mailing list > cur...@li... > https://lists.sourceforge.net/lists/listinfo/curl-loader-devel |
|
From: 邹靖晗 <zou...@gm...> - 2012-06-26 08:42:08
|
Hi all,
I am using curl-loader to try some network test, I use perl to drive
the test. I saved the general tags and a set of url tags in a hash table,
then use foreach to loop the hash table to output the configuration file
for curl-loader. So the order of tags is based on the hash method in perl.
Then I run curl-loader -f <the configuration file> and get a core dump.
Seems the order for the tags will impact the test result. So I want to
know what is the rule for the order of these tags in curl-loader?
Here is the configuration file:
[root@localhost jhzou]# cat 10.conf
CYCLES_NUM=-1
INTERFACE=eth1
CLIENTS_NUM_START=10
URLS_NUM=1
IP_ADDR_MIN=10.1.1.130
CLIENTS_RAMPUP_INC=5
BATCH_NAME=default_test
NETMASK=255.255.255.0
IP_ADDR_MAX=10.1.1.149
CLIENTS_NUM_MAX=20
TIMER_URL_COMPLETION=0
URL=http://10.1.1.122/index.htm
URL_SHORT_NAME=default_url
TIMER_AFTER_URL_SLEEP=0
REQUEST_TYPE=GET
This is the core dump debug informantion
Program received signal SIGSEGV, Segmentation fault.
timer_url_completion_parser (bctx=0x7ffffffe0b00, value=<value optimized
out>) at parse_conf.c:1899
1899
bctx->url_ctx_array[bctx->url_index].timer_url_completion_hrange =
(gdb) p bctx->url_index
$1 = 0
(gdb) p bctx->url_ctx_array[bctx->url_index]
Cannot access memory at address 0x0
(gdb) bt
#0 timer_url_completion_parser (bctx=0x7ffffffe0b00, value=<value
optimized out>) at parse_conf.c:1899
#1 0x000000000041c45f in add_param_to_batch (filename=<value optimized
out>, bctx_array=0x7ffffffe0b00,
bctx_array_size=64) at parse_conf.c:416
#2 parse_config_file (filename=<value optimized out>,
bctx_array=0x7ffffffe0b00, bctx_array_size=64)
at parse_conf.c:2897
#3 0x000000000040c524 in main (argc=<value optimized out>,
argv=0x7fffffffe588) at loader.c:159
--
Sincerely yours,
Zou Jinghan
|
|
From: Daniel A. J. <da...@ch...> - 2012-04-18 16:39:28
|
D'oh! Thank you. That, of course, fixed the problem :-) > Robert Iakobashvili <mailto:cor...@gm...> > Wednesday, April 18, 2012 1:45 AM > Hi Daniel, > > The IP-addresses that you're using should be "routable" within your > network, > > curl-loader adds these addresses to the interface specified, > and you can see them by: > > #ip addr. > > Let's say you have added addresses > > 10.10.0.1 - 10.10.0.100 > > Now you can test that they are routable to your destination IP > by : > > ping -I 10.10.0.1 Destination-IP > > where option -I (Ai-capital ) specifies the interface/IP to use for > testing. > > Thanks, > Robert > > > > > > > > > -- > Regards, > Robert Iakobashvili, Ph.D. > > Home: http://www.ghotit.com > Blog: http://www.ghotit.com/category/ghotit-blogs/ > Twitter: http://twitter.com/ghotit > Facebook: http://facebook.com/ghotit > ...................................................................... > Ghotit Dyslexia > Assistive technology that understands you > ...................................................................... > > ------------------------------------------------------------------------------ > Better than sec? Nothing is better than sec when it comes to > monitoring Big Data applications. Try Boundary one-second > resolution app monitoring today. Free. > http://p.sf.net/sfu/Boundary-dev2dev > _______________________________________________ > curl-loader-devel mailing list > cur...@li... > https://lists.sourceforge.net/lists/listinfo/curl-loader-devel > Daniel A. Jacobs <mailto:da...@ch...> > Tuesday, April 17, 2012 10:19 PM > > CURL-LOADER VERSION: version 0.56, unstable, January 10, 2012, svn 589 > > HW DETAILS: CPU/S and memory are must: > 2GB RAM, 2 cores of a Nehalem 2.2Ghz Xeon running under kvm with > plenty of spare capacity > > LINUX DISTRIBUTION and KERNEL (uname -r): > CentOS 6.2 > 2.6.32-220.7.1.el6.x86_64 > Running under kvm > > GCC VERSION (gcc -v): > Using built-in specs. > Target: x86_64-redhat-linux > Configured with: ../configure --prefix=/usr --mandir=/usr/share/man > --infodir=/usr/share/info > --with-bugurl=http://bugzilla.redhat.com/bugzilla --enable-bootstrap > --enable-shared --enable-threads=posix --enable-checking=release > --with-system-zlib --enable-__cxa_atexit > --disable-libunwind-exceptions --enable-gnu-unique-object > --enable-languages=c,c++,objc,obj-c++,java,fortran,ada > --enable-java-awt=gtk --disable-dssi > --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-1.5.0.0/jre > --enable-libgcj-multifile --enable-java-maintainer-mode > --with-ecj-jar=/usr/share/java/eclipse-ecj.jar > --disable-libjava-multilib --with-ppl --with-cloog --with-tune=generic > --with-arch_32=i686 --build=x86_64-redhat-linux > Thread model: posix > gcc version 4.4.6 20110731 (Red Hat 4.4.6-3) (GCC) > > > > COMPILATION AND MAKING OPTIONS (if defaults changed): > make optimize=1 debug=0 > > COMMAND-LINE: > curl-loader -f coloradotest.doortodoororganics.com > > CONFIGURATION-FILE (The most common source of problems): > Place the file inline here: > ########### GENERAL SECTION ################################ > > BATCH_NAME= coloradotest > CLIENTS_NUM_MAX=250 #Same as CLIENTS_NUM > CLIENTS_NUM_START=250 > CLIENTS_RAMPUP_INC=50 > INTERFACE =eth1 > NETMASK=16 > USER_AGENT="Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; > rv:1.9.0.1) Gecko/2008070208 Firefox/3.0.1" > IP_ADDR_MIN= 192.168.1.1 > IP_ADDR_MAX= 192.168.53.255 #Actually - this is for self-control > CYCLES_NUM= 50 > URLS_NUM = 1 > > ########### URL SECTION #################################### > > #URL= http://localhost/apache2-default/index.html > #http://localhost/index.html > URL=http://colorado.doortodoororganics.com/shop/veggies > #URL=http://localhost/ACE-INSTALL.html > REQUEST_TYPE=GET > TIMER_URL_COMPLETION = 0 # In msec. When positive, Now it is enforced > by cancelling url fetch on timeout > TIMER_AFTER_URL_SLEEP =1000 > > > DOES THE PROBLEM AFFECT: > COMPILATION? No > LINKING? No > EXECUTION? Yes > OTHER (please specify)? > Have you run $make cleanall prior to $make ? > Yes > > DESCRIPTION: > This is a VPS on the same network and same VM as the web server. Every > connection errors out, but I can get to the site fine with lynx, > traceroute, telnet, etc. (I have my DNS set up locally with a > different A record for colorado.doortodoororganics.com, as these are > staging servers on an internal 10.0.1 network.) > > I had curl-loader on the same physical machine as the web server and > it worked fine, but I really need to test it from a different machine. > I tried it from a different machine, running CentOS 5, on our internal > network as well, and got the same errors. > > # 1334721954169 Tue Apr 17 22:05:54 2012 > # msec_offset cycle_no url_no client_no (ip) indic info > 3120 0 0 241 !! ERR Failed connect to > colorado.doortodoororganics.com:80; No route to host > 3120 0 0 242 !! ERR Failed connect to > colorado.doortodoororganics.com:80; No route to host > 3120 0 0 243 !! ERR Failed connect to > colorado.doortodoororganics.com:80; No route to host > 5001 0 0 16 !! ERR Connection timed out after 5038 milliseconds > 5001 0 0 250 !! ERR Connection timed out after 5015 milliseconds > 5001 0 0 249 !! ERR Connection timed out after 5015 milliseconds > > > QUESTION/ SUGGESTION/ PATCH: > Any suggestions or pointers or solutions are most welcome! I really > need to get this testing done. Thank you. > > Daniel > > |
|
From: Robert I. <cor...@gm...> - 2012-04-18 07:45:29
|
Hi Daniel, The IP-addresses that you're using should be "routable" within your network, curl-loader adds these addresses to the interface specified, and you can see them by: #ip addr. Let's say you have added addresses 10.10.0.1 - 10.10.0.100 Now you can test that they are routable to your destination IP by : ping -I 10.10.0.1 Destination-IP where option -I (Ai-capital ) specifies the interface/IP to use for testing. Thanks, Robert On Wed, Apr 18, 2012 at 7:19 AM, Daniel A. Jacobs <da...@ch...>wrote: > > CURL-LOADER VERSION: version 0.56, unstable, January 10, 2012, svn 589 > > HW DETAILS: CPU/S and memory are must: > 2GB RAM, 2 cores of a Nehalem 2.2Ghz Xeon running under kvm with plenty > of spare capacity > > LINUX DISTRIBUTION and KERNEL (uname -r): > CentOS 6.2 > 2.6.32-220.7.1.el6.x86_64 > Running under kvm > > GCC VERSION (gcc -v): > Using built-in specs. > Target: x86_64-redhat-linux > Configured with: ../configure --prefix=/usr --mandir=/usr/share/man > --infodir=/usr/share/info > --with-bugurl=http://bugzilla.redhat.com/bugzilla --enable-bootstrap > --enable-shared --enable-threads=posix --enable-checking=release > --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions > --enable-gnu-unique-object > --enable-languages=c,c++,objc,obj-c++,java,fortran,ada > --enable-java-awt=gtk --disable-dssi > --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-1.5.0.0/jre > --enable-libgcj-multifile --enable-java-maintainer-mode > --with-ecj-jar=/usr/share/java/eclipse-ecj.jar > --disable-libjava-multilib --with-ppl --with-cloog --with-tune=generic > --with-arch_32=i686 --build=x86_64-redhat-linux > Thread model: posix > gcc version 4.4.6 20110731 (Red Hat 4.4.6-3) (GCC) > > > > COMPILATION AND MAKING OPTIONS (if defaults changed): > make optimize=1 debug=0 > > COMMAND-LINE: > curl-loader -f coloradotest.doortodoororganics.com > > CONFIGURATION-FILE (The most common source of problems): > Place the file inline here: > ########### GENERAL SECTION ################################ > > BATCH_NAME= coloradotest > CLIENTS_NUM_MAX=250 #Same as CLIENTS_NUM > CLIENTS_NUM_START=250 > CLIENTS_RAMPUP_INC=50 > INTERFACE =eth1 > NETMASK=16 > USER_AGENT="Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.1) > Gecko/2008070208 Firefox/3.0.1" > IP_ADDR_MIN= 192.168.1.1 > IP_ADDR_MAX= 192.168.53.255 #Actually - this is for self-control > CYCLES_NUM= 50 > URLS_NUM = 1 > > ########### URL SECTION #################################### > > #URL= http://localhost/apache2-default/index.html > #http://localhost/index.html > URL=http://colorado.doortodoororganics.com/shop/veggies > #URL=http://localhost/ACE-INSTALL.html > REQUEST_TYPE=GET > TIMER_URL_COMPLETION = 0 # In msec. When positive, Now it is enforced by > cancelling url fetch on timeout > TIMER_AFTER_URL_SLEEP =1000 > > > DOES THE PROBLEM AFFECT: > COMPILATION? No > LINKING? No > EXECUTION? Yes > OTHER (please specify)? > Have you run $make cleanall prior to $make ? > Yes > > DESCRIPTION: > This is a VPS on the same network and same VM as the web server. Every > connection errors out, but I can get to the site fine with lynx, > traceroute, telnet, etc. (I have my DNS set up locally with a different > A record for colorado.doortodoororganics.com, as these are staging > servers on an internal 10.0.1 network.) > > I had curl-loader on the same physical machine as the web server and it > worked fine, but I really need to test it from a different machine. I > tried it from a different machine, running CentOS 5, on our internal > network as well, and got the same errors. > > # 1334721954169 Tue Apr 17 22:05:54 2012 > # msec_offset cycle_no url_no client_no (ip) indic info > 3120 0 0 241 !! ERR Failed connect to > colorado.doortodoororganics.com:80; No route to host > 3120 0 0 242 !! ERR Failed connect to > colorado.doortodoororganics.com:80; No route to host > 3120 0 0 243 !! ERR Failed connect to > colorado.doortodoororganics.com:80; No route to host > 5001 0 0 16 !! ERR Connection timed out after 5038 milliseconds > 5001 0 0 250 !! ERR Connection timed out after 5015 milliseconds > 5001 0 0 249 !! ERR Connection timed out after 5015 milliseconds > > > QUESTION/ SUGGESTION/ PATCH: > Any suggestions or pointers or solutions are most welcome! I really need > to get this testing done. Thank you. > > Daniel > > > > > ------------------------------------------------------------------------------ > Better than sec? Nothing is better than sec when it comes to > monitoring Big Data applications. Try Boundary one-second > resolution app monitoring today. Free. > http://p.sf.net/sfu/Boundary-dev2dev > _______________________________________________ > curl-loader-devel mailing list > cur...@li... > https://lists.sourceforge.net/lists/listinfo/curl-loader-devel > -- Regards, Robert Iakobashvili, Ph.D. Home: http://www.ghotit.com Blog: http://www.ghotit.com/category/ghotit-blogs/ Twitter: http://twitter.com/ghotit Facebook: http://facebook.com/ghotit ...................................................................... Ghotit Dyslexia Assistive technology that understands you ...................................................................... |
|
From: Daniel A. J. <da...@ch...> - 2012-04-18 04:47:51
|
CURL-LOADER VERSION: version 0.56, unstable, January 10, 2012, svn 589 HW DETAILS: CPU/S and memory are must: 2GB RAM, 2 cores of a Nehalem 2.2Ghz Xeon running under kvm with plenty of spare capacity LINUX DISTRIBUTION and KERNEL (uname -r): CentOS 6.2 2.6.32-220.7.1.el6.x86_64 Running under kvm GCC VERSION (gcc -v): Using built-in specs. Target: x86_64-redhat-linux Configured with: ../configure --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --with-bugurl=http://bugzilla.redhat.com/bugzilla --enable-bootstrap --enable-shared --enable-threads=posix --enable-checking=release --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-gnu-unique-object --enable-languages=c,c++,objc,obj-c++,java,fortran,ada --enable-java-awt=gtk --disable-dssi --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-1.5.0.0/jre --enable-libgcj-multifile --enable-java-maintainer-mode --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --disable-libjava-multilib --with-ppl --with-cloog --with-tune=generic --with-arch_32=i686 --build=x86_64-redhat-linux Thread model: posix gcc version 4.4.6 20110731 (Red Hat 4.4.6-3) (GCC) COMPILATION AND MAKING OPTIONS (if defaults changed): make optimize=1 debug=0 COMMAND-LINE: curl-loader -f coloradotest.doortodoororganics.com CONFIGURATION-FILE (The most common source of problems): Place the file inline here: ########### GENERAL SECTION ################################ BATCH_NAME= coloradotest CLIENTS_NUM_MAX=250 #Same as CLIENTS_NUM CLIENTS_NUM_START=250 CLIENTS_RAMPUP_INC=50 INTERFACE =eth1 NETMASK=16 USER_AGENT="Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.1) Gecko/2008070208 Firefox/3.0.1" IP_ADDR_MIN= 192.168.1.1 IP_ADDR_MAX= 192.168.53.255 #Actually - this is for self-control CYCLES_NUM= 50 URLS_NUM = 1 ########### URL SECTION #################################### #URL= http://localhost/apache2-default/index.html #http://localhost/index.html URL=http://colorado.doortodoororganics.com/shop/veggies #URL=http://localhost/ACE-INSTALL.html REQUEST_TYPE=GET TIMER_URL_COMPLETION = 0 # In msec. When positive, Now it is enforced by cancelling url fetch on timeout TIMER_AFTER_URL_SLEEP =1000 DOES THE PROBLEM AFFECT: COMPILATION? No LINKING? No EXECUTION? Yes OTHER (please specify)? Have you run $make cleanall prior to $make ? Yes DESCRIPTION: This is a VPS on the same network and same VM as the web server. Every connection errors out, but I can get to the site fine with lynx, traceroute, telnet, etc. (I have my DNS set up locally with a different A record for colorado.doortodoororganics.com, as these are staging servers on an internal 10.0.1 network.) I had curl-loader on the same physical machine as the web server and it worked fine, but I really need to test it from a different machine. I tried it from a different machine, running CentOS 5, on our internal network as well, and got the same errors. # 1334721954169 Tue Apr 17 22:05:54 2012 # msec_offset cycle_no url_no client_no (ip) indic info 3120 0 0 241 !! ERR Failed connect to colorado.doortodoororganics.com:80; No route to host 3120 0 0 242 !! ERR Failed connect to colorado.doortodoororganics.com:80; No route to host 3120 0 0 243 !! ERR Failed connect to colorado.doortodoororganics.com:80; No route to host 5001 0 0 16 !! ERR Connection timed out after 5038 milliseconds 5001 0 0 250 !! ERR Connection timed out after 5015 milliseconds 5001 0 0 249 !! ERR Connection timed out after 5015 milliseconds QUESTION/ SUGGESTION/ PATCH: Any suggestions or pointers or solutions are most welcome! I really need to get this testing done. Thank you. Daniel |
|
From: Fred H. <di...@gm...> - 2012-03-30 03:29:59
|
Interval stats (latest:3 sec, clients:200, CAPS-curr:6): H/F Req:4851,1xx:4845,2xx:4838,3xx:0,4xx:0,5xx:0,Err:0,T-Err:0,D:6ms,D-2xx:20ms,Ti:549990B/s,To:117664880B/s H/F/S Req:0,1xx:0,2xx:0,3xx:0,4xx:0,5xx:0,Err:0,T-Err:0,D:0ms,D-2xx:0ms,Ti:0B/s,To:0B/s -------------------------------------------------------------------------------- Summary stats (runs:253 secs, CAPS-average:6): H/F Req:397722,1xx:397690,2xx:397571,3xx:0,4xx:0,5xx:0,Err:0,T-Err:0,D:2ms,D-2xx:14ms,Ti:535867B/s,To:111794260B/s H/F/S Req:0,1xx:0,2xx:0,3xx:0,4xx:0,5xx:0,Err:0,T-Err:0,D:0ms,D-2xx:0ms,Ti:0B/s,To:0B/s ================================================================================= Manual: clients:max[200],curr[200]. Inc num: [+|*]. ================================================================================= |
|
From: Robert I. <cor...@gm...> - 2012-03-07 15:09:04
|
I see. On Wed, Mar 7, 2012 at 3:49 PM, Fred Huang <di...@gm...> wrote: > I tried -t 16 or -t 32, the performance drops to approx. 50 CAPS... > > > 2012/3/7 Robert Iakobashvili <cor...@gm...> >> >> Fred, >> >> On more suggestion, which is a general one. >> >> If N is the number of your logical CPUs, >> usually, I'd recommend to run 2*N + B threads. >> >> Thus, you can try: >> >> -t 16 >> >> -t 32 >> >> and it MAY increased a bit CAPS. >> >> Thanks, >> Robert >> >> On Wed, Mar 7, 2012 at 2:43 AM, Fred Huang <di...@gm...> wrote: >> > yes, I use -t 8 option because the test box has 8 logic CPUs (Dell 990, >> > Intel Core i7 2600, 16GB memory). >> > if I do not use -t option, the CPU used by curl-loader will be 100% on >> > one >> > of 8 logic CPU (12% of total CPU), and the CAPS drops to 100: >> > # gprof /usr/bin/curl-loader gmon.out -p | head -20 >> > Flat profile: >> > Each sample counts as 0.01 seconds. >> > % cumulative self self total >> > time seconds seconds calls s/call s/call name >> > 34.12 2.74 2.74 17385 0.00 0.00 create_conn >> > 11.21 3.64 0.90 30998583 0.00 0.00 >> > hostcache_timestamp_remove >> > 9.84 4.43 0.79 13433 0.00 0.00 >> > Curl_hash_clean_with_criterium >> > 7.97 5.07 0.64 381818 0.00 0.00 Curl_hash_pick >> > 5.85 5.54 0.47 17385 0.00 0.00 ConnectionStore >> > 5.11 5.95 0.41 16385 0.00 0.00 >> > curl_multi_remove_handle >> > 4.11 6.28 0.33 16509889 0.00 0.00 >> > Curl_str_key_compare >> > 3.36 6.55 0.27 23959914 0.00 0.00 Curl_socket_check >> > 3.24 6.81 0.26 56703 0.00 0.00 Curl_hash_add >> > 2.62 7.02 0.21 635750 0.00 0.00 dprintf_formatf >> > 1.25 7.12 0.10 92421190 0.00 0.00 >> > ares__is_list_empty >> > 0.75 7.18 0.06 97957 0.00 0.00 Curl_readwrite >> > 0.62 7.23 0.05 23422260 0.00 0.00 addbyter >> > 0.62 7.28 0.05 1136775 0.00 0.00 >> > client_tracing_function >> > 0.50 7.32 0.04 3091375 0.00 0.00 >> > ares__init_list_head >> > >> > I also tried to use >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, 20); >> > but get no improvement on CAPS. >> > >> > >> > Regards, >> > Fred >> > >> > 2012/3/7 Robert Iakobashvili <cor...@gm...> >> >> >> >> Dear Fred, >> >> >> >> If I understand correctly your observations, the issue is not in: >> >> >> >> 1. DNS resolving; >> >> 2. Version of cares or libcurl; >> >> >> >> but rather in >> >> >> >> Less effective handling of HTTPS versus HTTP in libcurl. >> >> >> >> Please, confirm that my understanding is correct. >> >> >> >> >> >> HTTPS URLs are using an extra locking in openssl with -t option >> >> that could be making the issue. >> >> If you are using -t option, try to load without it and let's say with >> >> 300 clients. >> >> >> >> If you are not using -t, thus the only way to proceed is to share >> >> your observations with libcurl people at >> >> >> >> libcurl development <cur...@co...> >> >> >> >> Thank you. >> >> >> >> Best wishes. >> >> Robert >> >> >> >> On Tue, Mar 6, 2012 at 6:17 PM, Fred Huang <di...@gm...> wrote: >> >> > 1. the major CPU consumer is curl-loader. here is the top: >> >> > top - 07:36:36 up 23:31, 5 users, load average: 4.32, 2.26, 0.91 >> >> > Tasks: 237 total, 2 running, 235 sleeping, 0 stopped, 0 zombie >> >> > Cpu0 : 76.6%us, 3.6%sy, 0.0%ni, 19.4%id, 0.0%wa, 0.0%hi, >> >> > 0.3%si, >> >> > 0.0% >> >> > Cpu1 : 72.4%us, 4.0%sy, 0.0%ni, 22.9%id, 0.0%wa, 0.0%hi, >> >> > 0.7%si, >> >> > 0.0% >> >> > Cpu2 : 65.8%us, 3.3%sy, 0.0%ni, 30.6%id, 0.0%wa, 0.0%hi, >> >> > 0.3%si, >> >> > 0.0% >> >> > Cpu3 : 75.2%us, 4.3%sy, 0.0%ni, 19.5%id, 0.0%wa, 0.0%hi, >> >> > 1.0%si, >> >> > 0.0% >> >> > Cpu4 : 79.1%us, 4.0%sy, 0.0%ni, 16.8%id, 0.0%wa, 0.0%hi, >> >> > 0.0%si, >> >> > 0.0% >> >> > Cpu5 : 75.8%us, 4.3%sy, 0.0%ni, 19.2%id, 0.0%wa, 0.0%hi, >> >> > 0.7%si, >> >> > 0.0% >> >> > Cpu6 : 74.1%us, 5.0%sy, 0.0%ni, 20.3%id, 0.0%wa, 0.0%hi, >> >> > 0.7%si, >> >> > 0.0% >> >> > Cpu7 : 78.7%us, 4.0%sy, 0.0%ni, 16.6%id, 0.0%wa, 0.0%hi, >> >> > 0.7%si, >> >> > 0.0% >> >> > Mem: 16209932k total, 3829500k used, 12380432k free, 99112k >> >> > buffers >> >> > Swap: 18448376k total, 0k used, 18448376k free, 320284k >> >> > cached >> >> > >> >> > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND >> >> > 30795 root 20 0 3200m 2.8g 2296 S 79.2 18.4 21:26.84 >> >> > curl-loader >> >> > 1832 nobody 20 0 13944 1736 580 S 0.1 0.0 0:32.65 dnsmasq >> >> > 9 root 20 0 0 0 0 S 0.0 0.0 0:00.53 >> >> > ksoftirqd/1 >> >> > 30825 root 20 0 15220 1360 960 R 0.0 0.0 0:00.42 top >> >> > 1 root 20 0 19400 1568 1256 S 0.0 0.0 0:00.75 init >> >> > 2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd >> >> > 3 root RT 0 0 0 0 S 0.0 0.0 0:00.00 >> >> > migration/0 >> >> > 4 root 20 0 0 0 0 S 0.0 0.0 0:00.06 >> >> > ksoftirqd/0 >> >> > 5 root RT 0 0 0 0 S 0.0 0.0 0:00.00 >> >> > migration/0 >> >> > >> >> > >> >> > 2. 1000 loading clients >> >> > >> >> > 3. yes, tried 0.52 but no lucky -- CAPS 380, throughput 200Mbps, CPU >> >> > 90% >> >> > top - 08:17:10 up 1 day, 12 min, 8 users, load average: 4.66, 3.39, >> >> > 1.88 >> >> > Tasks: 245 total, 2 running, 243 sleeping, 0 stopped, 0 zombie >> >> > Cpu0 : 82.2%us, 4.0%sy, 0.0%ni, 13.9%id, 0.0%wa, 0.0%hi, >> >> > 0.0%si, >> >> > 0.0% >> >> > Cpu1 : 86.1%us, 4.3%sy, 0.0%ni, 7.9%id, 0.0%wa, 0.0%hi, >> >> > 1.7%si, >> >> > 0.0% >> >> > Cpu2 : 85.3%us, 4.0%sy, 0.0%ni, 10.0%id, 0.0%wa, 0.0%hi, >> >> > 0.7%si, >> >> > 0.0% >> >> > Cpu3 : 81.8%us, 3.0%sy, 0.0%ni, 14.2%id, 0.0%wa, 0.0%hi, >> >> > 1.0%si, >> >> > 0.0% >> >> > Cpu4 : 83.7%us, 4.0%sy, 0.0%ni, 12.3%id, 0.0%wa, 0.0%hi, >> >> > 0.0%si, >> >> > 0.0% >> >> > Cpu5 : 92.7%us, 3.3%sy, 0.0%ni, 3.3%id, 0.0%wa, 0.0%hi, >> >> > 0.7%si, >> >> > 0.0% >> >> > Cpu6 : 89.7%us, 2.0%sy, 0.0%ni, 8.0%id, 0.0%wa, 0.0%hi, >> >> > 0.3%si, >> >> > 0.0% >> >> > Cpu7 : 88.1%us, 3.6%sy, 0.0%ni, 6.6%id, 0.0%wa, 0.0%hi, >> >> > 1.7%si, >> >> > 0.0% >> >> > Mem: 16209932k total, 1795748k used, 14414184k free, 102824k >> >> > buffers >> >> > Swap: 18448376k total, 0k used, 18448376k free, 382988k >> >> > cached >> >> > >> >> > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND >> >> > 12236 root 20 0 1150m 894m 2572 S 90.7 5.7 51:11.91 >> >> > curl-loader >> >> > 1832 nobody 20 0 13944 1736 580 S 0.1 0.0 0:47.04 dnsmasq >> >> > >> >> > ]# gprof /usr/bin/curl-loader gmon.out -p | head -20 >> >> > Flat profile: >> >> > Each sample counts as 0.01 seconds. >> >> > % cumulative self self total >> >> > time seconds seconds calls s/call s/call name >> >> > 43.92 151.95 151.95 157295 0.00 0.00 >> >> > Curl_hash_clean_with_criterium >> >> > 22.90 231.17 79.22 1712190971 0.00 0.00 >> >> > hostcache_timestamp_remove >> >> > 13.15 276.68 45.51 3260321 0.00 0.00 Curl_hash_pick >> >> > 7.51 302.65 25.97 394171 0.00 0.00 Curl_hash_add >> >> > 5.98 323.34 20.68 567437518 0.00 0.00 >> >> > Curl_str_key_compare >> >> > 1.00 326.80 3.46 5380220 0.00 0.00 dprintf_formatf >> >> > 0.41 328.21 1.41 >> >> > locking_function >> >> > 0.32 329.32 1.11 469222138 0.00 0.00 >> >> > ares__is_list_empty >> >> > 0.31 330.40 1.08 3284508 0.00 0.00 multi_runsingle >> >> > 0.28 331.38 0.98 10006806 0.00 0.00 >> >> > client_tracing_function >> >> > 0.24 332.21 0.83 151951 0.00 0.00 >> >> > Curl_num_addresses >> >> > 0.22 332.97 0.77 188075672 0.00 0.00 addbyter >> >> > 0.19 333.64 0.67 841840 0.00 0.00 Curl_readwrite >> >> > 0.19 334.29 0.65 id_function >> >> > 0.18 334.90 0.61 39843 0.00 0.01 >> >> > curl_multi_socket_all >> >> > >> >> > >> >> > >> >> > 4. yes, it is correct for 0.56. If use same URL list by replacing >> >> > https >> >> > to >> >> > http, I can get CAPS 1500 and throughput 870Mbps with CPU% < 60% >> >> > >> >> > >> >> > 2012/3/6 Robert Iakobashvili <cor...@gm...> >> >> >> >> >> >> Several questions: >> >> >> >> >> >> 1. Could you please run top and see if there are other processes >> >> >> burning >> >> >> CPU? >> >> >> 2. What is the number of loading clients that you use? >> >> >> 3. Could you try curl-loader-0.52 if it compiles for you? >> >> >> If not, please try to place curl-loader-7.19.7.tar.gz and >> >> >> c-ares-16.0.tar.gz, >> >> >> update their versions in Makefile and rebuild by make cleanall; make >> >> >> >> >> >> 4. Is it correct to say that for HTTP URLs (not HTTPS) you are >> >> >> getting >> >> >> a good throughput >> >> >> and CAPS with curl-loader-0.56? >> >> >> >> >> >> Thanks, >> >> >> Robert >> >> >> >> >> >> On Tue, Mar 6, 2012 at 3:52 AM, Fred Huang <di...@gm...> wrote: >> >> >> > TIMEOUT -1 -> 60 works a little better but no performance >> >> >> > improvement >> >> >> > on >> >> >> > CAPS or throughput. >> >> >> > >> >> >> > test 5: >> >> >> > enable c-ares; >> >> >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, 60); >> >> >> > >> >> >> > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve >> >> >> > *.com >> >> >> > to >> >> >> > one IP address >> >> >> > number of domain names in URLs: 780,000 >> >> >> > number of client: 1000 >> >> >> > CPU usage: 74% >> >> >> > cpu% irq% sirq% sys% iowt% mem_used buf&cached >> >> >> > 74.0 0.0 0.7 4.2 0.0 3530.6Mb 250.8Mb >> >> >> > SSL TPS: 360 >> >> >> > SSL throughput: 220Mbps >> >> >> > >> >> >> > # gprof /usr/bin/curl-loader gmon.out -p | head -50 >> >> >> > Flat profile: >> >> >> > Each sample counts as 0.01 seconds. >> >> >> > % cumulative self self total >> >> >> > time seconds seconds calls s/call s/call name >> >> >> > 26.63 23.13 23.13 119226 0.00 0.00 >> >> >> > Curl_hash_clean_with_criterium >> >> >> > 20.34 40.80 17.67 317792146 0.00 0.00 >> >> >> > hostcache_timestamp_remove >> >> >> > 11.27 50.59 9.79 2579903 0.00 0.00 >> >> >> > Curl_hash_pick >> >> >> > 7.84 57.40 6.81 120606 0.00 0.00 create_conn >> >> >> > 5.65 62.31 4.91 119117401 0.00 0.00 >> >> >> > Curl_str_key_compare >> >> >> > 4.98 66.64 4.33 329829 0.00 0.00 >> >> >> > Curl_hash_add >> >> >> > 2.80 69.08 2.44 4476606 0.00 0.00 >> >> >> > dprintf_formatf >> >> >> > 2.57 71.31 2.23 124019 0.00 0.00 >> >> >> > ConnectionStore >> >> >> > 1.50 72.61 1.30 740997609 0.00 0.00 >> >> >> > ares__is_list_empty >> >> >> > 1.16 73.61 1.01 >> >> >> > locking_function >> >> >> > 0.76 74.27 0.66 238744 0.00 0.00 ares_cancel >> >> >> > 0.70 74.88 0.61 457564 0.00 0.00 >> >> >> > curl_multi_socket_action >> >> >> > 0.67 75.46 0.58 118616 0.00 0.00 >> >> >> > curl_multi_remove_handle >> >> >> > 0.58 75.97 0.51 8211502 0.00 0.00 >> >> >> > client_tracing_function >> >> >> > 0.53 76.43 0.46 733561 0.00 0.00 >> >> >> > Curl_readwrite >> >> >> > 0.51 76.87 0.44 2183179 0.00 0.00 >> >> >> > multi_runsingle >> >> >> > 0.51 77.31 0.44 166585567 0.00 0.00 addbyter >> >> >> > 0.44 77.69 0.38 2175805 0.00 0.00 >> >> >> > Curl_raw_equal >> >> >> > 0.43 78.06 0.38 id_function >> >> >> > 0.40 78.41 0.35 29789346 0.00 0.00 >> >> >> > Curl_socket_check >> >> >> > 0.40 78.76 0.35 479671 0.00 0.00 >> >> >> > ossl_connect_common >> >> >> > 0.40 79.11 0.35 490639 0.00 0.00 >> >> >> > ares__init_list_node >> >> >> > 0.33 79.40 0.29 94197670 0.00 0.00 >> >> >> > Curl_raw_toupper >> >> >> > 0.33 79.69 0.29 3268031 0.00 0.00 >> >> >> > Curl_pgrsUpdate >> >> >> > 0.28 79.93 0.24 8797 0.00 0.00 >> >> >> > curl_multi_perform >> >> >> > 0.27 80.17 0.24 2233641 0.00 0.00 Curl_infof >> >> >> > 0.26 80.40 0.23 1839837 0.00 0.00 Curl_splay >> >> >> > 0.24 80.61 0.21 121713028 0.00 0.00 >> >> >> > curl_strequal >> >> >> > 0.21 80.79 0.18 243333 0.00 0.00 >> >> >> > Curl_ssl_getsessionid >> >> >> > 0.18 80.95 0.16 122180 0.00 0.00 >> >> >> > Curl_num_addresses >> >> >> > 0.17 81.09 0.15 8271666 0.00 0.00 Curl_debug >> >> >> > 0.16 81.23 0.14 18902452 0.00 0.00 >> >> >> > alloc_addbyter >> >> >> > 0.16 81.37 0.14 8851 0.00 0.00 >> >> >> > curl_multi_socket_all >> >> >> > 0.14 81.50 0.13 2236769 0.00 0.00 Curl_read >> >> >> > 0.14 81.62 0.12 2271700 0.00 0.00 >> >> >> > fd_key_compare >> >> >> > 0.14 81.74 0.12 323167 0.00 0.00 event_del >> >> >> > 0.14 81.86 0.12 205436 0.00 0.00 >> >> >> > read_udp_packets >> >> >> > 0.14 81.98 0.12 200979 0.00 0.00 processfds >> >> >> > 0.13 82.09 0.11 123886 0.00 0.00 >> >> >> > Curl_if_is_interface_name >> >> >> > 0.13 82.20 0.11 120350 0.00 0.00 Curl_http >> >> >> > 0.13 82.31 0.11 >> >> >> > ssl_tls_trace >> >> >> > 0.12 82.41 0.10 119229 0.00 0.00 >> >> >> > Curl_splayremovebyaddr >> >> >> > 0.12 82.51 0.10 7798717 0.00 0.00 >> >> >> > Curl_raw_nequal >> >> >> > 0.12 82.61 0.10 784485 0.00 0.00 event_add >> >> >> > 0.10 82.70 0.09 643482 0.00 0.00 Curl_expire >> >> >> > >> >> >> > >> >> >> > >> >> >> > >> >> >> > >> >> >> > test 6: >> >> >> > enable c-ares; >> >> >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, 60); >> >> >> > CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.23.0 >> >> >> > >> >> >> > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve >> >> >> > *.com >> >> >> > to >> >> >> > one IP address >> >> >> > number of domain names in URLs: 780,000 >> >> >> > number of client: 1000 >> >> >> > CPU usage: 86% >> >> >> > cpu% irq% sirq% sys% iowt% mem_used buf&cached >> >> >> > 85.9 0.0 0.2 2.1 0.0 3537.2Mb 261.4Mb >> >> >> > SSL TPS: 360 >> >> >> > SSL throughput: 200Mbps >> >> >> > >> >> >> > # gprof /usr/bin/curl-loader gmon.out -p | head -20 >> >> >> > Flat profile: >> >> >> > Each sample counts as 0.01 seconds. >> >> >> > % cumulative self self total >> >> >> > time seconds seconds calls s/call s/call name >> >> >> > 27.19 19.96 19.96 103472 0.00 0.00 >> >> >> > Curl_hash_clean_with_criterium >> >> >> > 21.73 35.91 15.95 274168981 0.00 0.00 >> >> >> > hostcache_timestamp_remove >> >> >> > 10.33 43.49 7.58 2256441 0.00 0.00 >> >> >> > Curl_hash_pick >> >> >> > 7.21 48.78 5.29 104540 0.00 0.00 create_conn >> >> >> > 6.20 53.33 4.55 103413405 0.00 0.00 >> >> >> > Curl_str_key_compare >> >> >> > 4.88 56.91 3.58 287109 0.00 0.00 >> >> >> > Curl_hash_add >> >> >> > 3.02 59.13 2.22 3797855 0.00 0.00 >> >> >> > dprintf_formatf >> >> >> > 2.10 60.67 1.54 107711 0.00 0.00 >> >> >> > ConnectionStore >> >> >> > 1.65 61.88 1.21 642725327 0.00 0.00 >> >> >> > ares__is_list_empty >> >> >> > 1.19 62.76 0.88 >> >> >> > locking_function >> >> >> > 0.87 63.40 0.64 207817 0.00 0.00 ares_cancel >> >> >> > 0.56 63.81 0.41 404108 0.00 0.00 >> >> >> > curl_multi_socket_action >> >> >> > 0.54 64.21 0.40 425603 0.00 0.00 >> >> >> > ares__init_list_node >> >> >> > 0.53 64.60 0.39 7068112 0.00 0.00 >> >> >> > client_tracing_function >> >> >> > 0.49 64.96 0.36 1908587 0.00 0.00 >> >> >> > multi_runsingle >> >> >> > >> >> >> > >> >> >> > >> >> >> > test 7: >> >> >> > enable c-ares; >> >> >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, -1); >> >> >> > CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.23.0 >> >> >> > >> >> >> > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve >> >> >> > *.com >> >> >> > to >> >> >> > one IP address >> >> >> > number of domain names in URLs: 780,000 >> >> >> > number of client: 1000 >> >> >> > CPU usage: 86% >> >> >> > cpu% irq% sirq% sys% iowt% mem_used buf&cached >> >> >> > 86.6 0.0 0.6 3.3 0.0 3624.8Mb 263.0Mb >> >> >> > SSL TPS: 300 >> >> >> > SSL throughput: 200Mbps >> >> >> > >> >> >> > # gprof /usr/bin/curl-loader gmon.out -p | head -20 >> >> >> > Flat profile: >> >> >> > Each sample counts as 0.01 seconds. >> >> >> > % cumulative self self total >> >> >> > time seconds seconds calls s/call s/call name >> >> >> > 39.83 369.71 369.71 207173 0.00 0.00 >> >> >> > Curl_hash_clean_with_criterium >> >> >> > 24.27 594.98 225.27 3198337733 0.00 0.00 >> >> >> > hostcache_timestamp_remove >> >> >> > 13.75 722.59 127.61 4312864 0.00 0.00 >> >> >> > Curl_hash_pick >> >> >> > 9.19 807.87 85.28 546507 0.00 0.00 >> >> >> > Curl_hash_add >> >> >> > 6.14 864.88 57.01 1022419684 0.00 0.00 >> >> >> > Curl_str_key_compare >> >> >> > 2.06 884.00 19.12 203749 0.00 0.00 create_conn >> >> >> > 0.54 888.99 4.99 213306 0.00 0.00 >> >> >> > ConnectionStore >> >> >> > 0.49 893.50 4.51 7331251 0.00 0.00 >> >> >> > dprintf_formatf >> >> >> > 0.29 896.16 2.66 1287182329 0.00 0.00 >> >> >> > ares__is_list_empty >> >> >> > 0.23 898.28 2.12 >> >> >> > locking_function >> >> >> > 0.20 900.10 1.82 212210 0.00 0.00 >> >> >> > Curl_num_addresses >> >> >> > 0.17 901.66 1.56 198385 0.00 0.00 >> >> >> > curl_multi_remove_handle >> >> >> > 0.12 902.81 1.15 411114 0.00 0.00 ares_cancel >> >> >> > 0.12 903.95 1.14 3683448 0.00 0.00 >> >> >> > multi_runsingle >> >> >> > 0.11 904.97 1.03 13664120 0.00 0.00 >> >> >> > client_tracing_function >> >> >> > >> >> >> > >> >> >> > test 8: >> >> >> > enable c-ares; >> >> >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, 60); >> >> >> > CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.24.0 >> >> >> > >> >> >> > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve >> >> >> > *.com >> >> >> > to >> >> >> > one IP address >> >> >> > number of domain names in URLs: 780,000 >> >> >> > number of client: 1000 >> >> >> > CPU usage: 75% >> >> >> > cpu% irq% sirq% sys% iowt% mem_used buf&cached >> >> >> > 75.2 0.0 0.2 2.1 0.0 3537.2Mb 261.4Mb >> >> >> > SSL TPS: 380 >> >> >> > SSL throughput: 220Mbps >> >> >> > >> >> >> > # gprof /usr/bin/curl-loader gmon.out -p | head -20 >> >> >> > Flat profile: >> >> >> > Each sample counts as 0.01 seconds. >> >> >> > % cumulative self self total >> >> >> > time seconds seconds calls s/call s/call name >> >> >> > 27.58 12.31 12.31 71140 0.00 0.00 >> >> >> > Curl_hash_clean_with_criterium >> >> >> > 19.09 20.83 8.52 176729692 0.00 0.00 >> >> >> > hostcache_timestamp_remove >> >> >> > 10.44 25.49 4.66 1589309 0.00 0.00 >> >> >> > Curl_hash_pick >> >> >> > 6.85 28.55 3.06 73204 0.00 0.00 create_conn >> >> >> > 5.56 31.03 2.48 67620530 0.00 0.00 >> >> >> > Curl_str_key_compare >> >> >> > 4.41 33.00 1.97 200684 0.00 0.00 >> >> >> > Curl_hash_add >> >> >> > 3.56 34.59 1.59 2717956 0.00 0.00 >> >> >> > dprintf_formatf >> >> >> > 1.72 35.36 0.77 75247 0.00 0.00 >> >> >> > ConnectionStore >> >> >> > 1.70 36.12 0.76 444342256 0.00 0.00 >> >> >> > ares__is_list_empty >> >> >> > 1.23 36.67 0.55 >> >> >> > locking_function >> >> >> > 1.16 37.19 0.52 143342 0.00 0.00 ares_cancel >> >> >> > 0.99 37.63 0.44 72211 0.00 0.00 >> >> >> > curl_multi_remove_handle >> >> >> > 0.74 37.96 0.33 1333371 0.00 0.00 >> >> >> > multi_runsingle >> >> >> > 0.68 38.27 0.31 101147798 0.00 0.00 addbyter >> >> >> > 0.58 38.53 0.26 288737 0.00 0.00 >> >> >> > curl_multi_socket_action >> >> >> > >> >> >> > >> >> >> > >> >> >> > test 9: >> >> >> > enable c-ares; >> >> >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, -1); >> >> >> > CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.24.0 >> >> >> > >> >> >> > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve >> >> >> > *.com >> >> >> > to >> >> >> > one IP address >> >> >> > number of domain names in URLs: 780,000 >> >> >> > number of client: 1000 >> >> >> > CPU usage: 75% >> >> >> > cpu% irq% sirq% sys% iowt% mem_used buf&cached >> >> >> > 75.1 0.0 0.6 4.1 0.0 3545.2Mb 260.1Mb >> >> >> > SSL TPS: 380 >> >> >> > SSL throughput: 200Mbps >> >> >> > >> >> >> > # gprof /usr/bin/curl-loader gmon.out -p | head -20 >> >> >> > Flat profile: >> >> >> > Each sample counts as 0.01 seconds. >> >> >> > % cumulative self self total >> >> >> > time seconds seconds calls s/call s/call name >> >> >> > 34.00 31.13 31.13 72495 0.00 0.00 >> >> >> > Curl_hash_clean_with_criterium >> >> >> > 23.64 52.78 21.65 386070412 0.00 0.00 >> >> >> > hostcache_timestamp_remove >> >> >> > 13.28 64.94 12.16 1619440 0.00 0.00 >> >> >> > Curl_hash_pick >> >> >> > 5.92 70.36 5.42 137653129 0.00 0.00 >> >> >> > Curl_str_key_compare >> >> >> > 5.50 75.40 5.04 203018 0.00 0.00 >> >> >> > Curl_hash_add >> >> >> > 4.55 79.57 4.17 74618 0.00 0.00 create_conn >> >> >> > 1.70 81.13 1.56 2757148 0.00 0.00 >> >> >> > dprintf_formatf >> >> >> > 0.98 82.03 0.90 76945 0.00 0.00 >> >> >> > ConnectionStore >> >> >> > 0.95 82.90 0.87 455775100 0.00 0.00 >> >> >> > ares__is_list_empty >> >> >> > 0.57 83.42 0.53 >> >> >> > locking_function >> >> >> > 0.39 83.78 0.36 146742 0.00 0.00 ares_cancel >> >> >> > 0.35 84.10 0.32 102968806 0.00 0.00 addbyter >> >> >> > 0.35 84.42 0.32 456587 0.00 0.00 >> >> >> > Curl_readwrite >> >> >> > 0.35 84.74 0.32 73133 0.00 0.00 >> >> >> > curl_multi_remove_handle >> >> >> > 0.34 85.05 0.31 1360545 0.00 0.00 >> >> >> > multi_runsingle >> >> >> > >> >> >> > >> >> >> > 2012/3/6 Robert Iakobashvili <cor...@gm...> >> >> >> >> >> >> >> >> If it will not be helpful this means that recent >> >> >> >> changes in libcurl library have broken DNS interface. >> >> >> >> >> >> >> >> You can try a previos version and the latest libcurl, like: >> >> >> >> >> >> >> >> http://curl.haxx.se/download/curl-7.23.0.tar.gz >> >> >> >> and >> >> >> >> http://curl.haxx.se/downloadcurl-7.24.0.tar.gz >> >> >> >> >> >> >> >> by: >> >> >> >> >> >> >> >> 1. downloading a libcurl version and placing it to >> >> >> >> curl-loader-0.56/packages ; >> >> >> >> 2. correcting the Makefile in curl-loader-0.56 >> >> >> >> CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.24 or CURL_VER:=7.23 >> >> >> >> 3. make cleanall; make >> >> >> >> >> >> >> >> Please, send us your observations. >> >> >> >> Thanks! >> >> >> >> Robert >> >> >> >> >> >> >> >> On Mon, Mar 5, 2012 at 6:57 PM, Robert Iakobashvili >> >> >> >> <cor...@gm...> >> >> >> >> wrote: >> >> >> >> > If so, please try to change >> >> >> >> > >> >> >> >> > loader.c, line 578 which looks like: >> >> >> >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, -1); >> >> >> >> > >> >> >> >> > 1. instead of -1 try to set some value like 60 >> >> >> >> > 2. make clean; make >> >> >> >> > >> >> >> >> > Thanks, >> >> >> >> > Robert >> >> >> >> > >> >> >> >> > On Mon, Mar 5, 2012 at 6:19 PM, Fred Huang <di...@gm...> >> >> >> >> > wrote: >> >> >> >> >> >> >> >> >> >> Looks like the bottleneck comes from libcurl other than >> >> >> >> >> c-ares... >> >> >> >> >> test 3 (no ares patch applied): >> >> >> >> >> dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, >> >> >> >> >> resolve >> >> >> >> >> *.com >> >> >> >> >> to one IP address >> >> >> >> >> number of domain names in URLs: 780,000 >> >> >> >> >> number of client: 1000 >> >> >> >> >> CPU usage: 85% >> >> >> >> >> cpu% irq% sirq% sys% iowt% mem_used buf&cached >> >> >> >> >> 85.6 0.0 0.8 5.9 0.0 3334.3Mb 112.0Mb >> >> >> >> >> SSL TPS: 320 >> >> >> >> >> SSL throughput: 200Mbps >> >> >> >> >> >> >> >> >> >> # gprof /usr/bin/curl-loader gmon.out -p | head -50 >> >> >> >> >> Flat profile: >> >> >> >> >> Each sample counts as 0.01 seconds. >> >> >> >> >> % cumulative self self total >> >> >> >> >> time seconds seconds calls s/call s/call name >> >> >> >> >> 40.80 120.16 120.16 116079 0.00 0.00 >> >> >> >> >> Curl_hash_clean_with_criterium >> >> >> >> >> 24.46 192.18 72.02 983166323 0.00 0.00 >> >> >> >> >> hostcache_timestamp_remove >> >> >> >> >> 13.30 231.35 39.17 2059341 0.00 0.00 >> >> >> >> >> Curl_hash_pick >> >> >> >> >> 6.45 250.35 19.00 240087 0.00 0.00 >> >> >> >> >> Curl_hash_add >> >> >> >> >> 4.81 264.51 14.16 334257102 0.00 0.00 >> >> >> >> >> Curl_str_key_compare >> >> >> >> >> 3.40 274.52 10.01 119514 0.00 0.00 >> >> >> >> >> create_conn >> >> >> >> >> 0.93 277.25 2.73 4478910 0.00 0.00 >> >> >> >> >> dprintf_formatf >> >> >> >> >> 0.90 279.91 2.66 122209 0.00 0.00 >> >> >> >> >> ConnectionStore >> >> >> >> >> 0.40 281.09 1.18 >> >> >> >> >> locking_function >> >> >> >> >> 0.27 281.88 0.79 114759 0.00 0.00 >> >> >> >> >> curl_multi_remove_handle >> >> >> >> >> 0.21 282.51 0.63 7993664 0.00 0.00 >> >> >> >> >> client_tracing_function >> >> >> >> >> 0.21 283.12 0.61 122685 0.00 0.00 >> >> >> >> >> Curl_num_addresses >> >> >> >> >> 0.19 283.68 0.56 397028 0.00 0.00 >> >> >> >> >> curl_multi_socket_action >> >> >> >> >> 0.19 284.23 0.55 2036116 0.00 0.00 >> >> >> >> >> multi_runsingle >> >> >> >> >> 0.17 284.72 0.49 703687 0.00 0.00 >> >> >> >> >> Curl_readwrite >> >> >> >> >> 0.16 285.19 0.47 >> >> >> >> >> id_function >> >> >> >> >> 0.15 285.62 0.43 29692029 0.00 0.00 >> >> >> >> >> Curl_socket_check >> >> >> >> >> 0.14 286.02 0.40 91973856 0.00 0.00 >> >> >> >> >> Curl_raw_toupper >> >> >> >> >> 0.13 286.40 0.38 164664144 0.00 0.00 >> >> >> >> >> addbyter >> >> >> >> >> 0.13 286.77 0.37 2180797 0.00 0.00 >> >> >> >> >> Curl_infof >> >> >> >> >> 0.11 287.10 0.33 122675 0.00 0.00 >> >> >> >> >> Curl_if_is_interface_name >> >> >> >> >> 0.10 287.39 0.29 471318 0.00 0.00 >> >> >> >> >> ossl_connect_common >> >> >> >> >> 0.10 287.67 0.28 2113528 0.00 0.00 >> >> >> >> >> Curl_raw_equal >> >> >> >> >> 0.09 287.93 0.26 8676 0.00 0.02 >> >> >> >> >> curl_multi_socket_all >> >> >> >> >> 0.08 288.17 0.24 1663262 0.00 0.00 >> >> >> >> >> Curl_splay >> >> >> >> >> 0.08 288.41 0.24 8505 0.00 0.01 >> >> >> >> >> curl_multi_perform >> >> >> >> >> 0.07 288.63 0.22 3114576 0.00 0.00 >> >> >> >> >> Curl_pgrsUpdate >> >> >> >> >> 0.06 288.80 0.17 119637 0.00 0.00 >> >> >> >> >> singleipconnect >> >> >> >> >> 0.06 288.97 0.17 7994818 0.00 0.00 >> >> >> >> >> Curl_debug >> >> >> >> >> 0.05 289.13 0.16 235820 0.00 0.00 >> >> >> >> >> event_del >> >> >> >> >> 0.05 289.29 0.16 119393071 0.00 0.00 >> >> >> >> >> curl_strequal >> >> >> >> >> 0.05 289.45 0.16 18535473 0.00 0.00 >> >> >> >> >> alloc_addbyter >> >> >> >> >> 0.05 289.61 0.16 356074 0.00 0.00 >> >> >> >> >> Curl_hash_str >> >> >> >> >> 0.05 289.76 0.15 1797233 0.00 0.00 >> >> >> >> >> fd_key_compare >> >> >> >> >> 0.05 289.91 0.15 234966 0.00 0.00 >> >> >> >> >> Curl_ssl_getsessionid >> >> >> >> >> 0.04 290.04 0.13 585853 0.00 0.00 >> >> >> >> >> Curl_llist_destroy >> >> >> >> >> 0.04 290.16 0.12 240826 0.00 0.00 >> >> >> >> >> Curl_resolv >> >> >> >> >> 0.03 290.26 0.10 7514328 0.00 0.00 >> >> >> >> >> Curl_raw_nequal >> >> >> >> >> 0.03 290.36 0.10 2214436 0.00 0.00 >> >> >> >> >> Curl_read >> >> >> >> >> 0.03 290.46 0.10 116872 0.00 0.00 >> >> >> >> >> Curl_http_readwrite_headers >> >> >> >> >> 0.03 290.56 0.10 >> >> >> >> >> ssl_tls_trace >> >> >> >> >> 0.03 290.66 0.10 8013097 0.00 0.00 >> >> >> >> >> scan_response >> >> >> >> >> 0.03 290.75 0.09 29488555 0.00 0.00 >> >> >> >> >> curlx_tvdiff >> >> >> >> >> 0.03 290.84 0.09 2142468 0.00 0.00 >> >> >> >> >> Curl_setopt >> >> >> >> >> 0.03 290.93 0.09 476299 0.00 0.00 >> >> >> >> >> Curl_expire >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> 2012/3/5 Robert Iakobashvili <cor...@gm...> >> >> >> >> >>> >> >> >> >> >>> Dear Fred Huang, >> >> >> >> >>> >> >> >> >> >>> Thanks for your posting. >> >> >> >> >>> >> >> >> >> >>> Could you please check if this is due to some badness at >> >> >> >> >>> cares? >> >> >> >> >>> Thanks. >> >> >> >> >>> >> >> >> >> >>> Let's exclude usage of cares based DNS resolving by applying >> >> >> >> >>> the >> >> >> >> >>> patch >> >> >> >> >>> attached >> >> >> >> >>> and use instead global DNS cache. >> >> >> >> >>> >> >> >> >> >>> cd ./curl-loader-0.56 >> >> >> >> >>> patch -p1 < path-to/disable-cares-use-global-dns.patch >> >> >> >> >>> make cleanall; make >> >> >> >> >>> >> >> >> >> >>> See if it helps. >> >> >> >> >>> >> >> >> >> >>> Global cache, unfortunately, is not thread safe. >> >> >> >> >>> >> >> >> >> >>> If it helps, we'll try to restore the previous version of >> >> >> >> >>> cares. >> >> >> >> >>> >> >> >> >> >>> Thanks, >> >> >> >> >>> Robert >> >> >> >> >>> >> >> >> >> >>> On Mon, Mar 5, 2012 at 4:28 PM, Fred Huang <di...@gm...> >> >> >> >> >>> wrote: >> >> >> >> >>>> >> >> >> >> >>>> test 1: >> >> >> >> >>>> >> >> >> >> >>>> dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, >> >> >> >> >>>> resolve >> >> >> >> >>>> *.com to one IP address >> >> >> >> >>>> number of domain names in all https URLs: 780,000 >> >> >> >> >>>> number of client: 1000 >> >> >> >> >>>> CPU usage: 78% >> >> >> >> >>>> cpu% irq% sirq% sys% iowt% mem_used buf&cached >> >> >> >> >>>> 78.8 0.0 0.6 3.8 0.0 3640.9Mb 282.1Mb >> >> >> >> >>>> SSL TPS: 300 >> >> >> >> >>>> SSL throughput: 200Mbps >> >> >> >> >>>> >> >> >> >> >>>> # gprof /usr/bin/curl-loader gmon.out -p | head -50 >> >> >> >> >>>> Flat profile: >> >> >> >> >>>> Each sample counts as 0.01 seconds. >> >> >> >> >>>> % cumulative self self total >> >> >> >> >>>> time seconds seconds calls s/call s/call name >> >> >> >> >>>> 38.55 127.10 127.10 130975 0.00 0.00 >> >> >> >> >>>> Curl_hash_clean_with_criterium >> >> >> >> >>>> 24.63 208.32 81.22 1247528407 0.00 0.00 >> >> >> >> >>>> hostcache_timestamp_remove >> >> >> >> >>>> 13.08 251.44 43.12 2791988 0.00 0.00 >> >> >> >> >>>> Curl_hash_pick >> >> >> >> >>>> 7.30 275.51 24.07 352490 0.00 0.00 >> >> >> >> >>>> Curl_hash_add >> >> >> >> >>>> 6.01 295.31 19.80 410760770 0.00 0.00 >> >> >> >> >>>> Curl_str_key_compare >> >> >> >> >>>> 2.95 305.04 9.73 131015 0.00 0.00 >> >> >> >> >>>> create_conn >> >> >> >> >>>> 0.88 307.94 2.90 4841390 0.00 0.00 >> >> >> >> >>>> dprintf_formatf >> >> >> >> >>>> 0.74 310.38 2.44 136036 0.00 0.00 >> >> >> >> >>>> ConnectionStore >> >> >> >> >>>> 0.48 311.96 1.58 814604911 0.00 0.00 >> >> >> >> >>>> ares__is_list_empty >> >> >> >> >>>> 0.36 313.16 1.20 >> >> >> >> >>>> locking_function >> >> >> >> >>>> 0.27 314.06 0.90 261608 0.00 0.00 >> >> >> >> >>>> ares_cancel >> >> >> >> >>>> 0.22 314.79 0.73 127417 0.00 0.00 >> >> >> >> >>>> curl_multi_remove_handle >> >> >> >> >>>> 0.19 315.40 0.62 8906301 0.00 0.00 >> >> >> >> >>>> client_tracing_function >> >> >> >> >>>> 0.17 315.97 0.57 135305 0.00 0.00 >> >> >> >> >>>> Curl_num_addresses >> >> >> >> >>>> 0.17 316.54 0.57 181937907 0.00 0.00 >> >> >> >> >>>> addbyter >> >> >> >> >>>> 0.17 317.09 0.55 2374363 0.00 0.00 >> >> >> >> >>>> multi_runsingle >> >> >> >> >>>> 0.16 317.62 0.53 541954 0.00 0.00 >> >> >> >> >>>> ares__init_list_node >> >> >> >> >>>> 0.16 318.14 0.52 494412 0.00 0.00 >> >> >> >> >>>> curl_multi_socket_action >> >> >> >> >>>> 0.14 318.60 0.46 33578897 0.00 0.00 >> >> >> >> >>>> Curl_socket_check >> >> >> >> >>>> 0.13 319.04 0.44 102951052 0.00 0.00 >> >> >> >> >>>> Curl_raw_toupper >> >> >> >> >>>> 0.13 319.48 0.44 787558 0.00 0.00 >> >> >> >> >>>> Curl_readwrite >> >> >> >> >>>> 0.13 319.91 0.44 >> >> >> >> >>>> id_function >> >> >> >> >>>> 0.12 320.29 0.38 391390 0.00 0.00 >> >> >> >> >>>> Curl_hash_str >> >> >> >> >>>> 0.09 320.58 0.29 9216 0.00 0.00 >> >> >> >> >>>> curl_multi_perform >> >> >> >> >>>> 0.08 320.86 0.28 2350908 0.00 0.00 >> >> >> >> >>>> Curl_raw_equal >> >> >> >> >>>> 0.08 321.14 0.28 2019415 0.00 0.00 >> >> >> >> >>>> Curl_splay >> >> >> >> >>>> 0.08 321.40 0.26 520625 0.00 0.00 >> >> >> >> >>>> ossl_connect_common >> >> >> >> >>>> 0.08 321.65 0.25 3532992 0.00 0.00 >> >> >> >> >>>> Curl_pgrsUpdate >> >> >> >> >>>> 0.07 321.87 0.22 131778537 0.00 0.00 >> >> >> >> >>>> curl_strequal >> >> >> >> >>>> 0.05 322.05 0.18 8954118 0.00 0.00 >> >> >> >> >>>> scan_response >> >> >> >> >>>> 0.05 322.23 0.18 533644 0.00 0.00 >> >> >> >> >>>> ares_expand_name >> >> >> >> >>>> 0.05 322.40 0.17 2477345 0.00 0.00 >> >> >> >> >>>> fd_key_compare >> >> >> >> >>>> 0.05 322.57 0.17 2429135 0.00 0.00 >> >> >> >> >>>> Curl_infof >> >> >> >> >>>> 0.05 322.74 0.17 130966 0.00 0.00 >> >> >> >> >>>> singleipconnect >> >> >> >> >>>> 0.05 322.90 0.16 9292 0.00 0.00 >> >> >> >> >>>> curl_multi_socket_all >> >> >> >> >>>> 0.05 323.06 0.16 270137 0.00 0.00 >> >> >> >> >>>> ares__get_hostent >> >> >> >> >>>> 0.05 323.21 0.15 8943657 0.00 0.00 >> >> >> >> >>>> Curl_debug >> >> >> >> >>>> 0.04 323.35 0.14 262493 0.00 0.00 >> >> >> >> >>>> Curl_ssl_getsessionid >> >> >> >> >>>> 0.04 323.48 0.13 566006 0.00 0.00 >> >> >> >> >>>> socket_callback >> >> >> >> >>>> 0.04 323.61 0.13 33983864 0.00 0.00 >> >> >> >> >>>> curlx_tvdiff >> >> >> >> >>>> 0.04 323.74 0.13 132853 0.00 0.00 >> >> >> >> >>>> Curl_http_readwrite_headers >> >> >> >> >>>> 0.04 323.86 0.12 344032 0.00 0.00 >> >> >> >> >>>> epoll_del >> >> >> >> >>>> 0.04 323.98 0.12 223477 0.00 0.00 >> >> >> >> >>>> Curl_poll >> >> >> >> >>>> 0.04 324.10 0.12 8526280 0.00 0.00 >> >> >> >> >>>> Curl_raw_nequal >> >> >> >> >>>> 0.03 324.21 0.11 348839 0.00 0.00 >> >> >> >> >>>> event_del >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >>>> test 2: >> >> >> >> >>>> >> >> >> >> >>>> dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, >> >> >> >> >>>> resolve >> >> >> >> >>>> *.com to one IP address >> >> >> >> >>>> number of domain names in all https URLs: 1 >> >> >> >> >>>> number of client: 1000 >> >> >> >> >>>> CPU usage: 75% >> >> >> >> >>>> SSL TPS: 1300 >> >> >> >> >>>> SSL throughput: 700Mbps >> >> >> >> >>>> >> >> >> >> >>>> # gprof /usr/bin/curl-loader gmon.out -p | head -50 >> >> >> >> >>>> Flat profile: >> >> >> >> >>>> Each sample counts as 0.01 seconds. >> >> >> >> >>>> % cumulative self self total >> >> >> >> >>>> time seconds seconds calls s/call s/call name >> >> >> >> >>>> 8.98 2.45 2.45 5127312 0.00 0.00 >> >> >> >> >>>> dprintf_formatf >> >> >> >> >>>> 8.90 4.88 2.43 1292796503 0.00 0.00 >> >> >> >> >>>> ares__is_list_empty >> >> >> >> >>>> 6.12 6.55 1.67 322461 0.00 0.00 >> >> >> >> >>>> create_conn >> >> >> >> >>>> 4.58 7.80 1.25 419872 0.00 0.00 >> >> >> >> >>>> ares_cancel >> >> >> >> >>>> 4.10 8.92 1.12 2100162 0.00 0.00 >> >> >> >> >>>> Curl_readwrite >> >> >> >> >>>> 3.92 9.99 1.07 4527388 0.00 0.00 >> >> >> >> >>>> Curl_hash_pick >> >> >> >> >>>> 3.43 10.93 0.94 >> >> >> >> >>>> locking_function >> >> >> >> >>>> 3.04 11.76 0.83 909759 0.00 0.00 >> >> >> >> >>>> curl_multi_socket_action >> >> >> >> >>>> 3.00 12.58 0.82 97252 0.00 0.00 >> >> >> >> >>>> Curl_hash_clean_with_criterium >> >> >> >> >>>> 2.71 13.32 0.74 179704016 0.00 0.00 >> >> >> >> >>>> Curl_raw_toupper >> >> >> >> >>>> 2.68 14.05 0.73 4646524 0.00 0.00 >> >> >> >> >>>> multi_runsingle >> >> >> >> >>>> 2.44 14.71 0.67 13862992 0.00 0.00 >> >> >> >> >>>> client_tracing_function >> >> >> >> >>>> 2.42 15.37 0.66 319735 0.00 0.00 >> >> >> >> >>>> curl_multi_remove_handle >> >> >> >> >>>> 2.42 16.03 0.66 22 0.03 0.03 >> >> >> >> >>>> ares__init_list_node >> >> >> >> >>>> 1.94 16.56 0.53 9930244 0.00 0.00 >> >> >> >> >>>> hostcache_timestamp_remove >> >> >> >> >>>> 1.94 17.09 0.53 169320088 0.00 0.00 >> >> >> >> >>>> addbyter >> >> >> >> >>>> 1.80 17.58 0.49 >> >> >> >> >>>> id_function >> >> >> >> >>>> 1.36 17.95 0.37 7057146 0.00 0.00 >> >> >> >> >>>> Curl_pgrsUpdate >> >> >> >> >>>> 1.36 18.32 0.37 16835 0.00 0.00 >> >> >> >> >>>> curl_multi_socket_all >> >> >> >> >>>> 1.25 18.66 0.34 16719 0.00 0.00 >> >> >> >> >>>> curl_multi_perform >> >> >> >> >>>> 1.21 18.99 0.33 329403 0.00 0.00 >> >> >> >> >>>> Curl_http_readwrite_headers >> >> >> >> >>>> 1.17 19.31 0.32 4036624 0.00 0.00 >> >> >> >> >>>> Curl_splay >> >> >> >> >>>> 1.14 19.62 0.31 20838902 0.00 0.00 >> >> >> >> >>>> Curl_raw_nequal >> >> >> >> >>>> 0.82 19.85 0.23 2764123 0.00 0.00 >> >> >> >> >>>> Curl_infof >> >> >> >> >>>> 0.81 20.07 0.22 5933857 0.00 0.00 >> >> >> >> >>>> ossl_recv >> >> >> >> >>>> 0.77 20.28 0.21 323112 0.00 0.00 >> >> >> >> >>>> Curl_splayremovebyaddr >> >> >> >> >>>> 0.75 20.48 0.21 5943991 0.00 0.00 >> >> >> >> >>>> Curl_read >> >> >> >> >>>> 0.70 20.67 0.19 425174 0.00 0.00 >> >> >> >> >>>> event_del >> >> >> >> >>>> 0.70 20.86 0.19 98652 0.00 0.00 >> >> >> >> >>>> Curl_if_is_interface_name >> >> >> >> >>>> 0.70 21.05 0.19 8677733 0.00 0.00 >> >> >> >> >>>> Curl_socket_check >> >> >> >> >>>> 0.59 21.21 0.16 4129554 0.00 0.00 >> >> >> >> >>>> fd_key_compare >> >> >> >> >>>> 0.57 21.37 0.16 14017019 0.00 0.00 >> >> >> >> >>>> Curl_debug >> >> >> >> >>>> 0.55 21.52 0.15 9376861 0.00 0.00 >> >> >> >> >>>> stat_data_in_add >> >> >> >> >>>> 0.55 21.67 0.15 6840468 0.00 0.00 >> >> >> >> >>>> Curl_timeleft >> >> >> >> >>>> 0.55 21.82 0.15 2054602 0.00 0.00 >> >> >> >> >>>> Curl_raw_equal >> >> >> >> >>>> 0.55 21.97 0.15 1291611 0.00 0.00 >> >> >> >> >>>> Curl_expire >> >> >> >> >>>> 0.51 22.11 0.14 16570063 0.00 0.00 >> >> >> >> >>>> curlx_tvnow >> >> >> >> >>>> 0.51 22.25 0.14 13863284 0.00 0.00 >> >> >> >> >>>> scan_response >> >> >> >> >>>> 0.51 22.39 0.14 5498864 0.00 0.00 >> >> >> >> >>>> Curl_setopt >> >> >> >> >>>> 0.48 22.52 0.13 97490291 0.00 0.00 >> >> >> >> >>>> curl_strequal >> >> >> >> >>>> 0.44 22.64 0.12 324253 0.00 0.00 >> >> >> >> >>>> Curl_http >> >> >> >> >>>> 0.44 22.76 0.12 394386 0.00 0.00 >> >> >> >> >>>> ossl_connect_common >> >> >> >> >>>> 0.40 22.87 0.11 3581335 0.00 0.00 >> >> >> >> >>>> Curl_getinfo >> >> >> >> >>>> 0.38 22.97 0.11 26983092 0.00 0.00 >> >> >> >> >>>> alloc_addbyter >> >> >> >> >>>> 0.37 23.07 0.10 8729044 0.00 0.00 >> >> >> >> >>>> Curl_client_write >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >>>> ------------------------------------------------------------------------------ >> >> >> >> >>>> Try before you buy = See our experts in action! >> >> >> >> >>>> The most comprehensive online learning library for Microsoft >> >> >> >> >>>> developers >> >> >> >> >>>> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, >> >> >> >> >>>> CSS3, >> >> >> >> >>>> MVC3, >> >> >> >> >>>> Metro Style Apps, more. Free future releases when you >> >> >> >> >>>> subscribe >> >> >> >> >>>> now! >> >> >> >> >>>> http://p.sf.net/sfu/learndevnow-dev2 >> >> >> >> >>>> _______________________________________________ >> >> >> >> >>>> curl-loader-devel mailing list >> >> >> >> >>>> cur...@li... >> >> >> >> >>>> >> >> >> >> >>>> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel >> >> >> >> >>>> >> >> >> >> >>> >> >> >> >> >>> >> >> >> >> >>> >> >> >> >> >>> -- >> >> >> >> >>> Regards, >> >> >> >> >>> Robert Iakobashvili, Ph.D. >> >> >> >> >>> >> >> >> >> >>> Home: http://www.ghotit.com >> >> >> >> >>> Blog: http://www.ghotit.com/category/ghotit-blogs/ >> >> >> >> >>> Twitter: http://twitter.com/ghotit >> >> >> >> >>> Facebook: http://facebook.com/ghotit >> >> >> >> >>> >> >> >> >> >>> >> >> >> >> >>> >> >> >> >> >>> ...................................................................... >> >> >> >> >>> Ghotit Dyslexia >> >> >> >> >>> Assistive technology that understands you >> >> >> >> >>> >> >> >> >> >>> >> >> >> >> >>> >> >> >> >> >>> ...................................................................... >> >> >> >> >>> >> >> >> >> >>> >> >> >> >> >>> >> >> >> >> >>> >> >> >> >> >>> >> >> >> >> >>> >> >> >> >> >>> ------------------------------------------------------------------------------ >> >> >> >> >>> Try before you buy = See our experts in action! >> >> >> >> >>> The most comprehensive online learning library for Microsoft >> >> >> >> >>> developers >> >> >> >> >>> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, >> >> >> >> >>> CSS3, >> >> >> >> >>> MVC3, >> >> >> >> >>> Metro Style Apps, more. Free future releases when you >> >> >> >> >>> subscribe >> >> >> >> >>> now! >> >> >> >> >>> http://p.sf.net/sfu/learndevnow-dev2 >> >> >> >> >>> _______________________________________________ >> >> >> >> >>> curl-loader-devel mailing list >> >> >> >> >>> cur...@li... >> >> >> >> >>> >> >> >> >> >>> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel >> >> >> >> >>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> ------------------------------------------------------------------------------ >> >> >> >> >> Try before you buy = See our experts in action! >> >> >> >> >> The most comprehensive online learning library for Microsoft >> >> >> >> >> developers >> >> >> >> >> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, >> >> >> >> >> CSS3, >> >> >> >> >> MVC3, >> >> >> >> >> Metro Style Apps, more. Free future releases when you >> >> >> >> >> subscribe >> >> >> >> >> now! >> >> >> >> >> http://p.sf.net/sfu/learndevnow-dev2 >> >> >> >> >> _______________________________________________ >> >> >> >> >> curl-loader-devel mailing list >> >> >> >> >> cur...@li... >> >> >> >> >> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel >> >> >> >> >> >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> > -- >> >> >> >> > Regards, >> >> >> >> > Robert Iakobashvili, Ph.D. >> >> >> >> > >> >> >> >> > Home: http://www.ghotit.com >> >> >> >> > Blog: http://www.ghotit.com/category/ghotit-blogs/ >> >> >> >> > Twitter: http://twitter.com/ghotit >> >> >> >> > Facebook: http://facebook.com/ghotit >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> > ...................................................................... >> >> >> >> > Ghotit Dyslexia >> >> >> >> > Assistive technology that understands you >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> > ...................................................................... >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> -- >> >> >> >> Regards, >> >> >> >> Robert Iakobashvili, Ph.D. >> >> >> >> >> >> >> >> Home: http://www.ghotit.com >> >> >> >> Blog: http://www.ghotit.com/category/ghotit-blogs/ >> >> >> >> Twitter: http://twitter.com/ghotit >> >> >> >> Facebook: http://facebook.com/ghotit >> >> >> >> >> >> >> >> >> >> >> >> ...................................................................... >> >> >> >> Ghotit Dyslexia >> >> >> >> Assistive technology that understands you >> >> >> >> >> >> >> >> >> >> >> >> ...................................................................... >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> ------------------------------------------------------------------------------ >> >> >> >> Try before you buy = See our experts in action! >> >> >> >> The most comprehensive online learning library for Microsoft >> >> >> >> developers >> >> >> >> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, >> >> >> >> CSS3, >> >> >> >> MVC3, >> >> >> >> Metro Style Apps, more. Free future releases when you subscribe >> >> >> >> now! >> >> >> >> http://p.sf.net/sfu/learndevnow-dev2 >> >> >> >> _______________________________________________ >> >> >> >> curl-loader-devel mailing list >> >> >> >> cur...@li... >> >> >> >> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel >> >> >> > >> >> >> > >> >> >> > >> >> >> > >> >> >> > >> >> >> > >> >> >> > ------------------------------------------------------------------------------ >> >> >> > Keep Your Developer Skills Current with LearnDevNow! >> >> >> > The most comprehensive online learning library for Microsoft >> >> >> > developers >> >> >> > is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, >> >> >> > MVC3, >> >> >> > Metro Style Apps, more. Free future releases when you subscribe >> >> >> > now! >> >> >> > http://p.sf.net/sfu/learndevnow-d2d >> >> >> > _______________________________________________ >> >> >> > curl-loader-devel mailing list >> >> >> > cur...@li... >> >> >> > https://lists.sourceforge.net/lists/listinfo/curl-loader-devel >> >> >> > >> >> >> >> >> >> >> >> >> >> >> >> -- >> >> >> Regards, >> >> >> Robert Iakobashvili, Ph.D. >> >> >> >> >> >> Home: http://www.ghotit.com >> >> >> Blog: http://www.ghotit.com/category/ghotit-blogs/ >> >> >> Twitter: http://twitter.com/ghotit >> >> >> Facebook: http://facebook.com/ghotit >> >> >> >> >> >> ...................................................................... >> >> >> Ghotit Dyslexia >> >> >> Assistive technology that understands you >> >> >> >> >> >> ...................................................................... >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> ------------------------------------------------------------------------------ >> >> >> Keep Your Developer Skills Current with LearnDevNow! >> >> >> The most comprehensive online learning library for Microsoft >> >> >> developers >> >> >> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, >> >> >> MVC3, >> >> >> Metro Style Apps, more. Free future releases when you subscribe now! >> >> >> http://p.sf.net/sfu/learndevnow-d2d >> >> >> _______________________________________________ >> >> >> curl-loader-devel mailing list >> >> >> cur...@li... >> >> >> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > ------------------------------------------------------------------------------ >> >> > Keep Your Developer Skills Current with LearnDevNow! >> >> > The most comprehensive online learning library for Microsoft >> >> > developers >> >> > is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, >> >> > MVC3, >> >> > Metro Style Apps, more. Free future releases when you subscribe now! >> >> > http://p.sf.net/sfu/learndevnow-d2d >> >> > _______________________________________________ >> >> > curl-loader-devel mailing list >> >> > cur...@li... >> >> > https://lists.sourceforge.net/lists/listinfo/curl-loader-devel >> >> > >> >> >> >> >> >> >> >> -- >> >> Regards, >> >> Robert Iakobashvili, Ph.D. >> >> >> >> Home: http://www.ghotit.com >> >> Blog: http://www.ghotit.com/category/ghotit-blogs/ >> >> Twitter: http://twitter.com/ghotit >> >> Facebook: http://facebook.com/ghotit >> >> ...................................................................... >> >> Ghotit Dyslexia >> >> Assistive technology that understands you >> >> ...................................................................... >> >> >> >> >> >> >> >> ------------------------------------------------------------------------------ >> >> Keep Your Developer Skills Current with LearnDevNow! >> >> The most comprehensive online learning library for Microsoft developers >> >> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, >> >> MVC3, >> >> Metro Style Apps, more. Free future releases when you subscribe now! >> >> http://p.sf.net/sfu/learndevnow-d2d >> >> _______________________________________________ >> >> curl-loader-devel mailing list >> >> cur...@li... >> >> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel >> > >> > >> > >> > >> > ------------------------------------------------------------------------------ >> > Virtualization & Cloud Management Using Capacity Planning >> > Cloud computing makes use of virtualization - but cloud computing >> > also focuses on allowing computing to be delivered as a service. >> > http://www.accelacomm.com/jaw/sfnl/114/51521223/ >> > _______________________________________________ >> > curl-loader-devel mailing list >> > cur...@li... >> > https://lists.sourceforge.net/lists/listinfo/curl-loader-devel >> > >> >> >> >> -- >> Regards, >> Robert Iakobashvili, Ph.D. >> >> Home: http://www.ghotit.com >> Blog: http://www.ghotit.com/category/ghotit-blogs/ >> Twitter: http://twitter.com/ghotit >> Facebook: http://facebook.com/ghotit >> ...................................................................... >> Ghotit Dyslexia >> Assistive technology that understands you >> ...................................................................... >> >> >> ------------------------------------------------------------------------------ >> Virtualization & Cloud Management Using Capacity Planning >> Cloud computing makes use of virtualization - but cloud computing >> also focuses on allowing computing to be delivered as a service. >> http://www.accelacomm.com/jaw/sfnl/114/51521223/ >> _______________________________________________ >> curl-loader-devel mailing list >> cur...@li... >> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel > > > > ------------------------------------------------------------------------------ > Virtualization & Cloud Management Using Capacity Planning > Cloud computing makes use of virtualization - but cloud computing > also focuses on allowing computing to be delivered as a service. > http://www.accelacomm.com/jaw/sfnl/114/51521223/ > _______________________________________________ > curl-loader-devel mailing list > cur...@li... > https://lists.sourceforge.net/lists/listinfo/curl-loader-devel > -- Regards, Robert Iakobashvili, Ph.D. Home: http://www.ghotit.com Blog: http://www.ghotit.com/category/ghotit-blogs/ Twitter: http://twitter.com/ghotit Facebook: http://facebook.com/ghotit ...................................................................... Ghotit Dyslexia Assistive technology that understands you ...................................................................... |
|
From: Fred H. <di...@gm...> - 2012-03-07 13:49:19
|
I tried -t 16 or -t 32, the performance drops to approx. 50 CAPS... 2012/3/7 Robert Iakobashvili <cor...@gm...> > Fred, > > On more suggestion, which is a general one. > > If N is the number of your logical CPUs, > usually, I'd recommend to run 2*N + B threads. > > Thus, you can try: > > -t 16 > > -t 32 > > and it MAY increased a bit CAPS. > > Thanks, > Robert > > On Wed, Mar 7, 2012 at 2:43 AM, Fred Huang <di...@gm...> wrote: > > yes, I use -t 8 option because the test box has 8 logic CPUs (Dell 990, > > Intel Core i7 2600, 16GB memory). > > if I do not use -t option, the CPU used by curl-loader will be 100% on > one > > of 8 logic CPU (12% of total CPU), and the CAPS drops to 100: > > # gprof /usr/bin/curl-loader gmon.out -p | head -20 > > Flat profile: > > Each sample counts as 0.01 seconds. > > % cumulative self self total > > time seconds seconds calls s/call s/call name > > 34.12 2.74 2.74 17385 0.00 0.00 create_conn > > 11.21 3.64 0.90 30998583 0.00 0.00 > > hostcache_timestamp_remove > > 9.84 4.43 0.79 13433 0.00 0.00 > > Curl_hash_clean_with_criterium > > 7.97 5.07 0.64 381818 0.00 0.00 Curl_hash_pick > > 5.85 5.54 0.47 17385 0.00 0.00 ConnectionStore > > 5.11 5.95 0.41 16385 0.00 0.00 > > curl_multi_remove_handle > > 4.11 6.28 0.33 16509889 0.00 0.00 > Curl_str_key_compare > > 3.36 6.55 0.27 23959914 0.00 0.00 Curl_socket_check > > 3.24 6.81 0.26 56703 0.00 0.00 Curl_hash_add > > 2.62 7.02 0.21 635750 0.00 0.00 dprintf_formatf > > 1.25 7.12 0.10 92421190 0.00 0.00 ares__is_list_empty > > 0.75 7.18 0.06 97957 0.00 0.00 Curl_readwrite > > 0.62 7.23 0.05 23422260 0.00 0.00 addbyter > > 0.62 7.28 0.05 1136775 0.00 0.00 > > client_tracing_function > > 0.50 7.32 0.04 3091375 0.00 0.00 > ares__init_list_head > > > > I also tried to use > > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, 20); > > but get no improvement on CAPS. > > > > > > Regards, > > Fred > > > > 2012/3/7 Robert Iakobashvili <cor...@gm...> > >> > >> Dear Fred, > >> > >> If I understand correctly your observations, the issue is not in: > >> > >> 1. DNS resolving; > >> 2. Version of cares or libcurl; > >> > >> but rather in > >> > >> Less effective handling of HTTPS versus HTTP in libcurl. > >> > >> Please, confirm that my understanding is correct. > >> > >> > >> HTTPS URLs are using an extra locking in openssl with -t option > >> that could be making the issue. > >> If you are using -t option, try to load without it and let's say with > >> 300 clients. > >> > >> If you are not using -t, thus the only way to proceed is to share > >> your observations with libcurl people at > >> > >> libcurl development <cur...@co...> > >> > >> Thank you. > >> > >> Best wishes. > >> Robert > >> > >> On Tue, Mar 6, 2012 at 6:17 PM, Fred Huang <di...@gm...> wrote: > >> > 1. the major CPU consumer is curl-loader. here is the top: > >> > top - 07:36:36 up 23:31, 5 users, load average: 4.32, 2.26, 0.91 > >> > Tasks: 237 total, 2 running, 235 sleeping, 0 stopped, 0 zombie > >> > Cpu0 : 76.6%us, 3.6%sy, 0.0%ni, 19.4%id, 0.0%wa, 0.0%hi, 0.3%si, > >> > 0.0% > >> > Cpu1 : 72.4%us, 4.0%sy, 0.0%ni, 22.9%id, 0.0%wa, 0.0%hi, 0.7%si, > >> > 0.0% > >> > Cpu2 : 65.8%us, 3.3%sy, 0.0%ni, 30.6%id, 0.0%wa, 0.0%hi, 0.3%si, > >> > 0.0% > >> > Cpu3 : 75.2%us, 4.3%sy, 0.0%ni, 19.5%id, 0.0%wa, 0.0%hi, 1.0%si, > >> > 0.0% > >> > Cpu4 : 79.1%us, 4.0%sy, 0.0%ni, 16.8%id, 0.0%wa, 0.0%hi, 0.0%si, > >> > 0.0% > >> > Cpu5 : 75.8%us, 4.3%sy, 0.0%ni, 19.2%id, 0.0%wa, 0.0%hi, 0.7%si, > >> > 0.0% > >> > Cpu6 : 74.1%us, 5.0%sy, 0.0%ni, 20.3%id, 0.0%wa, 0.0%hi, 0.7%si, > >> > 0.0% > >> > Cpu7 : 78.7%us, 4.0%sy, 0.0%ni, 16.6%id, 0.0%wa, 0.0%hi, 0.7%si, > >> > 0.0% > >> > Mem: 16209932k total, 3829500k used, 12380432k free, 99112k > buffers > >> > Swap: 18448376k total, 0k used, 18448376k free, 320284k > cached > >> > > >> > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > >> > 30795 root 20 0 3200m 2.8g 2296 S 79.2 18.4 21:26.84 > curl-loader > >> > 1832 nobody 20 0 13944 1736 580 S 0.1 0.0 0:32.65 dnsmasq > >> > 9 root 20 0 0 0 0 S 0.0 0.0 0:00.53 > ksoftirqd/1 > >> > 30825 root 20 0 15220 1360 960 R 0.0 0.0 0:00.42 top > >> > 1 root 20 0 19400 1568 1256 S 0.0 0.0 0:00.75 init > >> > 2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd > >> > 3 root RT 0 0 0 0 S 0.0 0.0 0:00.00 > migration/0 > >> > 4 root 20 0 0 0 0 S 0.0 0.0 0:00.06 > ksoftirqd/0 > >> > 5 root RT 0 0 0 0 S 0.0 0.0 0:00.00 > migration/0 > >> > > >> > > >> > 2. 1000 loading clients > >> > > >> > 3. yes, tried 0.52 but no lucky -- CAPS 380, throughput 200Mbps, CPU > 90% > >> > top - 08:17:10 up 1 day, 12 min, 8 users, load average: 4.66, 3.39, > >> > 1.88 > >> > Tasks: 245 total, 2 running, 243 sleeping, 0 stopped, 0 zombie > >> > Cpu0 : 82.2%us, 4.0%sy, 0.0%ni, 13.9%id, 0.0%wa, 0.0%hi, 0.0%si, > >> > 0.0% > >> > Cpu1 : 86.1%us, 4.3%sy, 0.0%ni, 7.9%id, 0.0%wa, 0.0%hi, 1.7%si, > >> > 0.0% > >> > Cpu2 : 85.3%us, 4.0%sy, 0.0%ni, 10.0%id, 0.0%wa, 0.0%hi, 0.7%si, > >> > 0.0% > >> > Cpu3 : 81.8%us, 3.0%sy, 0.0%ni, 14.2%id, 0.0%wa, 0.0%hi, 1.0%si, > >> > 0.0% > >> > Cpu4 : 83.7%us, 4.0%sy, 0.0%ni, 12.3%id, 0.0%wa, 0.0%hi, 0.0%si, > >> > 0.0% > >> > Cpu5 : 92.7%us, 3.3%sy, 0.0%ni, 3.3%id, 0.0%wa, 0.0%hi, 0.7%si, > >> > 0.0% > >> > Cpu6 : 89.7%us, 2.0%sy, 0.0%ni, 8.0%id, 0.0%wa, 0.0%hi, 0.3%si, > >> > 0.0% > >> > Cpu7 : 88.1%us, 3.6%sy, 0.0%ni, 6.6%id, 0.0%wa, 0.0%hi, 1.7%si, > >> > 0.0% > >> > Mem: 16209932k total, 1795748k used, 14414184k free, 102824k > buffers > >> > Swap: 18448376k total, 0k used, 18448376k free, 382988k > cached > >> > > >> > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > >> > 12236 root 20 0 1150m 894m 2572 S 90.7 5.7 51:11.91 > curl-loader > >> > 1832 nobody 20 0 13944 1736 580 S 0.1 0.0 0:47.04 dnsmasq > >> > > >> > ]# gprof /usr/bin/curl-loader gmon.out -p | head -20 > >> > Flat profile: > >> > Each sample counts as 0.01 seconds. > >> > % cumulative self self total > >> > time seconds seconds calls s/call s/call name > >> > 43.92 151.95 151.95 157295 0.00 0.00 > >> > Curl_hash_clean_with_criterium > >> > 22.90 231.17 79.22 1712190971 0.00 0.00 > >> > hostcache_timestamp_remove > >> > 13.15 276.68 45.51 3260321 0.00 0.00 Curl_hash_pick > >> > 7.51 302.65 25.97 394171 0.00 0.00 Curl_hash_add > >> > 5.98 323.34 20.68 567437518 0.00 0.00 > >> > Curl_str_key_compare > >> > 1.00 326.80 3.46 5380220 0.00 0.00 dprintf_formatf > >> > 0.41 328.21 1.41 locking_function > >> > 0.32 329.32 1.11 469222138 0.00 0.00 > >> > ares__is_list_empty > >> > 0.31 330.40 1.08 3284508 0.00 0.00 multi_runsingle > >> > 0.28 331.38 0.98 10006806 0.00 0.00 > >> > client_tracing_function > >> > 0.24 332.21 0.83 151951 0.00 0.00 > Curl_num_addresses > >> > 0.22 332.97 0.77 188075672 0.00 0.00 addbyter > >> > 0.19 333.64 0.67 841840 0.00 0.00 Curl_readwrite > >> > 0.19 334.29 0.65 id_function > >> > 0.18 334.90 0.61 39843 0.00 0.01 > >> > curl_multi_socket_all > >> > > >> > > >> > > >> > 4. yes, it is correct for 0.56. If use same URL list by replacing > https > >> > to > >> > http, I can get CAPS 1500 and throughput 870Mbps with CPU% < 60% > >> > > >> > > >> > 2012/3/6 Robert Iakobashvili <cor...@gm...> > >> >> > >> >> Several questions: > >> >> > >> >> 1. Could you please run top and see if there are other processes > >> >> burning > >> >> CPU? > >> >> 2. What is the number of loading clients that you use? > >> >> 3. Could you try curl-loader-0.52 if it compiles for you? > >> >> If not, please try to place curl-loader-7.19.7.tar.gz and > >> >> c-ares-16.0.tar.gz, > >> >> update their versions in Makefile and rebuild by make cleanall; make > >> >> > >> >> 4. Is it correct to say that for HTTP URLs (not HTTPS) you are > getting > >> >> a good throughput > >> >> and CAPS with curl-loader-0.56? > >> >> > >> >> Thanks, > >> >> Robert > >> >> > >> >> On Tue, Mar 6, 2012 at 3:52 AM, Fred Huang <di...@gm...> wrote: > >> >> > TIMEOUT -1 -> 60 works a little better but no performance > improvement > >> >> > on > >> >> > CAPS or throughput. > >> >> > > >> >> > test 5: > >> >> > enable c-ares; > >> >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, 60); > >> >> > > >> >> > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve > >> >> > *.com > >> >> > to > >> >> > one IP address > >> >> > number of domain names in URLs: 780,000 > >> >> > number of client: 1000 > >> >> > CPU usage: 74% > >> >> > cpu% irq% sirq% sys% iowt% mem_used buf&cached > >> >> > 74.0 0.0 0.7 4.2 0.0 3530.6Mb 250.8Mb > >> >> > SSL TPS: 360 > >> >> > SSL throughput: 220Mbps > >> >> > > >> >> > # gprof /usr/bin/curl-loader gmon.out -p | head -50 > >> >> > Flat profile: > >> >> > Each sample counts as 0.01 seconds. > >> >> > % cumulative self self total > >> >> > time seconds seconds calls s/call s/call name > >> >> > 26.63 23.13 23.13 119226 0.00 0.00 > >> >> > Curl_hash_clean_with_criterium > >> >> > 20.34 40.80 17.67 317792146 0.00 0.00 > >> >> > hostcache_timestamp_remove > >> >> > 11.27 50.59 9.79 2579903 0.00 0.00 > Curl_hash_pick > >> >> > 7.84 57.40 6.81 120606 0.00 0.00 create_conn > >> >> > 5.65 62.31 4.91 119117401 0.00 0.00 > >> >> > Curl_str_key_compare > >> >> > 4.98 66.64 4.33 329829 0.00 0.00 Curl_hash_add > >> >> > 2.80 69.08 2.44 4476606 0.00 0.00 > dprintf_formatf > >> >> > 2.57 71.31 2.23 124019 0.00 0.00 > ConnectionStore > >> >> > 1.50 72.61 1.30 740997609 0.00 0.00 > >> >> > ares__is_list_empty > >> >> > 1.16 73.61 1.01 > >> >> > locking_function > >> >> > 0.76 74.27 0.66 238744 0.00 0.00 ares_cancel > >> >> > 0.70 74.88 0.61 457564 0.00 0.00 > >> >> > curl_multi_socket_action > >> >> > 0.67 75.46 0.58 118616 0.00 0.00 > >> >> > curl_multi_remove_handle > >> >> > 0.58 75.97 0.51 8211502 0.00 0.00 > >> >> > client_tracing_function > >> >> > 0.53 76.43 0.46 733561 0.00 0.00 > Curl_readwrite > >> >> > 0.51 76.87 0.44 2183179 0.00 0.00 > multi_runsingle > >> >> > 0.51 77.31 0.44 166585567 0.00 0.00 addbyter > >> >> > 0.44 77.69 0.38 2175805 0.00 0.00 > Curl_raw_equal > >> >> > 0.43 78.06 0.38 id_function > >> >> > 0.40 78.41 0.35 29789346 0.00 0.00 > >> >> > Curl_socket_check > >> >> > 0.40 78.76 0.35 479671 0.00 0.00 > >> >> > ossl_connect_common > >> >> > 0.40 79.11 0.35 490639 0.00 0.00 > >> >> > ares__init_list_node > >> >> > 0.33 79.40 0.29 94197670 0.00 0.00 > >> >> > Curl_raw_toupper > >> >> > 0.33 79.69 0.29 3268031 0.00 0.00 > Curl_pgrsUpdate > >> >> > 0.28 79.93 0.24 8797 0.00 0.00 > >> >> > curl_multi_perform > >> >> > 0.27 80.17 0.24 2233641 0.00 0.00 Curl_infof > >> >> > 0.26 80.40 0.23 1839837 0.00 0.00 Curl_splay > >> >> > 0.24 80.61 0.21 121713028 0.00 0.00 > curl_strequal > >> >> > 0.21 80.79 0.18 243333 0.00 0.00 > >> >> > Curl_ssl_getsessionid > >> >> > 0.18 80.95 0.16 122180 0.00 0.00 > >> >> > Curl_num_addresses > >> >> > 0.17 81.09 0.15 8271666 0.00 0.00 Curl_debug > >> >> > 0.16 81.23 0.14 18902452 0.00 0.00 > alloc_addbyter > >> >> > 0.16 81.37 0.14 8851 0.00 0.00 > >> >> > curl_multi_socket_all > >> >> > 0.14 81.50 0.13 2236769 0.00 0.00 Curl_read > >> >> > 0.14 81.62 0.12 2271700 0.00 0.00 > fd_key_compare > >> >> > 0.14 81.74 0.12 323167 0.00 0.00 event_del > >> >> > 0.14 81.86 0.12 205436 0.00 0.00 > >> >> > read_udp_packets > >> >> > 0.14 81.98 0.12 200979 0.00 0.00 processfds > >> >> > 0.13 82.09 0.11 123886 0.00 0.00 > >> >> > Curl_if_is_interface_name > >> >> > 0.13 82.20 0.11 120350 0.00 0.00 Curl_http > >> >> > 0.13 82.31 0.11 ssl_tls_trace > >> >> > 0.12 82.41 0.10 119229 0.00 0.00 > >> >> > Curl_splayremovebyaddr > >> >> > 0.12 82.51 0.10 7798717 0.00 0.00 > Curl_raw_nequal > >> >> > 0.12 82.61 0.10 784485 0.00 0.00 event_add > >> >> > 0.10 82.70 0.09 643482 0.00 0.00 Curl_expire > >> >> > > >> >> > > >> >> > > >> >> > > >> >> > > >> >> > test 6: > >> >> > enable c-ares; > >> >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, 60); > >> >> > CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.23.0 > >> >> > > >> >> > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve > >> >> > *.com > >> >> > to > >> >> > one IP address > >> >> > number of domain names in URLs: 780,000 > >> >> > number of client: 1000 > >> >> > CPU usage: 86% > >> >> > cpu% irq% sirq% sys% iowt% mem_used buf&cached > >> >> > 85.9 0.0 0.2 2.1 0.0 3537.2Mb 261.4Mb > >> >> > SSL TPS: 360 > >> >> > SSL throughput: 200Mbps > >> >> > > >> >> > # gprof /usr/bin/curl-loader gmon.out -p | head -20 > >> >> > Flat profile: > >> >> > Each sample counts as 0.01 seconds. > >> >> > % cumulative self self total > >> >> > time seconds seconds calls s/call s/call name > >> >> > 27.19 19.96 19.96 103472 0.00 0.00 > >> >> > Curl_hash_clean_with_criterium > >> >> > 21.73 35.91 15.95 274168981 0.00 0.00 > >> >> > hostcache_timestamp_remove > >> >> > 10.33 43.49 7.58 2256441 0.00 0.00 > Curl_hash_pick > >> >> > 7.21 48.78 5.29 104540 0.00 0.00 create_conn > >> >> > 6.20 53.33 4.55 103413405 0.00 0.00 > >> >> > Curl_str_key_compare > >> >> > 4.88 56.91 3.58 287109 0.00 0.00 Curl_hash_add > >> >> > 3.02 59.13 2.22 3797855 0.00 0.00 > dprintf_formatf > >> >> > 2.10 60.67 1.54 107711 0.00 0.00 > ConnectionStore > >> >> > 1.65 61.88 1.21 642725327 0.00 0.00 > >> >> > ares__is_list_empty > >> >> > 1.19 62.76 0.88 > >> >> > locking_function > >> >> > 0.87 63.40 0.64 207817 0.00 0.00 ares_cancel > >> >> > 0.56 63.81 0.41 404108 0.00 0.00 > >> >> > curl_multi_socket_action > >> >> > 0.54 64.21 0.40 425603 0.00 0.00 > >> >> > ares__init_list_node > >> >> > 0.53 64.60 0.39 7068112 0.00 0.00 > >> >> > client_tracing_function > >> >> > 0.49 64.96 0.36 1908587 0.00 0.00 > multi_runsingle > >> >> > > >> >> > > >> >> > > >> >> > test 7: > >> >> > enable c-ares; > >> >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, -1); > >> >> > CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.23.0 > >> >> > > >> >> > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve > >> >> > *.com > >> >> > to > >> >> > one IP address > >> >> > number of domain names in URLs: 780,000 > >> >> > number of client: 1000 > >> >> > CPU usage: 86% > >> >> > cpu% irq% sirq% sys% iowt% mem_used buf&cached > >> >> > 86.6 0.0 0.6 3.3 0.0 3624.8Mb 263.0Mb > >> >> > SSL TPS: 300 > >> >> > SSL throughput: 200Mbps > >> >> > > >> >> > # gprof /usr/bin/curl-loader gmon.out -p | head -20 > >> >> > Flat profile: > >> >> > Each sample counts as 0.01 seconds. > >> >> > % cumulative self self total > >> >> > time seconds seconds calls s/call s/call name > >> >> > 39.83 369.71 369.71 207173 0.00 0.00 > >> >> > Curl_hash_clean_with_criterium > >> >> > 24.27 594.98 225.27 3198337733 0.00 0.00 > >> >> > hostcache_timestamp_remove > >> >> > 13.75 722.59 127.61 4312864 0.00 0.00 > Curl_hash_pick > >> >> > 9.19 807.87 85.28 546507 0.00 0.00 Curl_hash_add > >> >> > 6.14 864.88 57.01 1022419684 0.00 0.00 > >> >> > Curl_str_key_compare > >> >> > 2.06 884.00 19.12 203749 0.00 0.00 create_conn > >> >> > 0.54 888.99 4.99 213306 0.00 0.00 > ConnectionStore > >> >> > 0.49 893.50 4.51 7331251 0.00 0.00 > dprintf_formatf > >> >> > 0.29 896.16 2.66 1287182329 0.00 0.00 > >> >> > ares__is_list_empty > >> >> > 0.23 898.28 2.12 > >> >> > locking_function > >> >> > 0.20 900.10 1.82 212210 0.00 0.00 > >> >> > Curl_num_addresses > >> >> > 0.17 901.66 1.56 198385 0.00 0.00 > >> >> > curl_multi_remove_handle > >> >> > 0.12 902.81 1.15 411114 0.00 0.00 ares_cancel > >> >> > 0.12 903.95 1.14 3683448 0.00 0.00 > multi_runsingle > >> >> > 0.11 904.97 1.03 13664120 0.00 0.00 > >> >> > client_tracing_function > >> >> > > >> >> > > >> >> > test 8: > >> >> > enable c-ares; > >> >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, 60); > >> >> > CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.24.0 > >> >> > > >> >> > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve > >> >> > *.com > >> >> > to > >> >> > one IP address > >> >> > number of domain names in URLs: 780,000 > >> >> > number of client: 1000 > >> >> > CPU usage: 75% > >> >> > cpu% irq% sirq% sys% iowt% mem_used buf&cached > >> >> > 75.2 0.0 0.2 2.1 0.0 3537.2Mb 261.4Mb > >> >> > SSL TPS: 380 > >> >> > SSL throughput: 220Mbps > >> >> > > >> >> > # gprof /usr/bin/curl-loader gmon.out -p | head -20 > >> >> > Flat profile: > >> >> > Each sample counts as 0.01 seconds. > >> >> > % cumulative self self total > >> >> > time seconds seconds calls s/call s/call name > >> >> > 27.58 12.31 12.31 71140 0.00 0.00 > >> >> > Curl_hash_clean_with_criterium > >> >> > 19.09 20.83 8.52 176729692 0.00 0.00 > >> >> > hostcache_timestamp_remove > >> >> > 10.44 25.49 4.66 1589309 0.00 0.00 > Curl_hash_pick > >> >> > 6.85 28.55 3.06 73204 0.00 0.00 create_conn > >> >> > 5.56 31.03 2.48 67620530 0.00 0.00 > >> >> > Curl_str_key_compare > >> >> > 4.41 33.00 1.97 200684 0.00 0.00 Curl_hash_add > >> >> > 3.56 34.59 1.59 2717956 0.00 0.00 > dprintf_formatf > >> >> > 1.72 35.36 0.77 75247 0.00 0.00 > ConnectionStore > >> >> > 1.70 36.12 0.76 444342256 0.00 0.00 > >> >> > ares__is_list_empty > >> >> > 1.23 36.67 0.55 > >> >> > locking_function > >> >> > 1.16 37.19 0.52 143342 0.00 0.00 ares_cancel > >> >> > 0.99 37.63 0.44 72211 0.00 0.00 > >> >> > curl_multi_remove_handle > >> >> > 0.74 37.96 0.33 1333371 0.00 0.00 > multi_runsingle > >> >> > 0.68 38.27 0.31 101147798 0.00 0.00 addbyter > >> >> > 0.58 38.53 0.26 288737 0.00 0.00 > >> >> > curl_multi_socket_action > >> >> > > >> >> > > >> >> > > >> >> > test 9: > >> >> > enable c-ares; > >> >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, -1); > >> >> > CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.24.0 > >> >> > > >> >> > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve > >> >> > *.com > >> >> > to > >> >> > one IP address > >> >> > number of domain names in URLs: 780,000 > >> >> > number of client: 1000 > >> >> > CPU usage: 75% > >> >> > cpu% irq% sirq% sys% iowt% mem_used buf&cached > >> >> > 75.1 0.0 0.6 4.1 0.0 3545.2Mb 260.1Mb > >> >> > SSL TPS: 380 > >> >> > SSL throughput: 200Mbps > >> >> > > >> >> > # gprof /usr/bin/curl-loader gmon.out -p | head -20 > >> >> > Flat profile: > >> >> > Each sample counts as 0.01 seconds. > >> >> > % cumulative self self total > >> >> > time seconds seconds calls s/call s/call name > >> >> > 34.00 31.13 31.13 72495 0.00 0.00 > >> >> > Curl_hash_clean_with_criterium > >> >> > 23.64 52.78 21.65 386070412 0.00 0.00 > >> >> > hostcache_timestamp_remove > >> >> > 13.28 64.94 12.16 1619440 0.00 0.00 > Curl_hash_pick > >> >> > 5.92 70.36 5.42 137653129 0.00 0.00 > >> >> > Curl_str_key_compare > >> >> > 5.50 75.40 5.04 203018 0.00 0.00 Curl_hash_add > >> >> > 4.55 79.57 4.17 74618 0.00 0.00 create_conn > >> >> > 1.70 81.13 1.56 2757148 0.00 0.00 > dprintf_formatf > >> >> > 0.98 82.03 0.90 76945 0.00 0.00 > ConnectionStore > >> >> > 0.95 82.90 0.87 455775100 0.00 0.00 > >> >> > ares__is_list_empty > >> >> > 0.57 83.42 0.53 > >> >> > locking_function > >> >> > 0.39 83.78 0.36 146742 0.00 0.00 ares_cancel > >> >> > 0.35 84.10 0.32 102968806 0.00 0.00 addbyter > >> >> > 0.35 84.42 0.32 456587 0.00 0.00 > Curl_readwrite > >> >> > 0.35 84.74 0.32 73133 0.00 0.00 > >> >> > curl_multi_remove_handle > >> >> > 0.34 85.05 0.31 1360545 0.00 0.00 > multi_runsingle > >> >> > > >> >> > > >> >> > 2012/3/6 Robert Iakobashvili <cor...@gm...> > >> >> >> > >> >> >> If it will not be helpful this means that recent > >> >> >> changes in libcurl library have broken DNS interface. > >> >> >> > >> >> >> You can try a previos version and the latest libcurl, like: > >> >> >> > >> >> >> http://curl.haxx.se/download/curl-7.23.0.tar.gz > >> >> >> and > >> >> >> http://curl.haxx.se/downloadcurl-7.24.0.tar.gz > >> >> >> > >> >> >> by: > >> >> >> > >> >> >> 1. downloading a libcurl version and placing it to > >> >> >> curl-loader-0.56/packages ; > >> >> >> 2. correcting the Makefile in curl-loader-0.56 > >> >> >> CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.24 or CURL_VER:=7.23 > >> >> >> 3. make cleanall; make > >> >> >> > >> >> >> Please, send us your observations. > >> >> >> Thanks! > >> >> >> Robert > >> >> >> > >> >> >> On Mon, Mar 5, 2012 at 6:57 PM, Robert Iakobashvili > >> >> >> <cor...@gm...> > >> >> >> wrote: > >> >> >> > If so, please try to change > >> >> >> > > >> >> >> > loader.c, line 578 which looks like: > >> >> >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, -1); > >> >> >> > > >> >> >> > 1. instead of -1 try to set some value like 60 > >> >> >> > 2. make clean; make > >> >> >> > > >> >> >> > Thanks, > >> >> >> > Robert > >> >> >> > > >> >> >> > On Mon, Mar 5, 2012 at 6:19 PM, Fred Huang <di...@gm...> > >> >> >> > wrote: > >> >> >> >> > >> >> >> >> Looks like the bottleneck comes from libcurl other than > c-ares... > >> >> >> >> test 3 (no ares patch applied): > >> >> >> >> dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, > resolve > >> >> >> >> *.com > >> >> >> >> to one IP address > >> >> >> >> number of domain names in URLs: 780,000 > >> >> >> >> number of client: 1000 > >> >> >> >> CPU usage: 85% > >> >> >> >> cpu% irq% sirq% sys% iowt% mem_used buf&cached > >> >> >> >> 85.6 0.0 0.8 5.9 0.0 3334.3Mb 112.0Mb > >> >> >> >> SSL TPS: 320 > >> >> >> >> SSL throughput: 200Mbps > >> >> >> >> > >> >> >> >> # gprof /usr/bin/curl-loader gmon.out -p | head -50 > >> >> >> >> Flat profile: > >> >> >> >> Each sample counts as 0.01 seconds. > >> >> >> >> % cumulative self self total > >> >> >> >> time seconds seconds calls s/call s/call name > >> >> >> >> 40.80 120.16 120.16 116079 0.00 0.00 > >> >> >> >> Curl_hash_clean_with_criterium > >> >> >> >> 24.46 192.18 72.02 983166323 0.00 0.00 > >> >> >> >> hostcache_timestamp_remove > >> >> >> >> 13.30 231.35 39.17 2059341 0.00 0.00 > >> >> >> >> Curl_hash_pick > >> >> >> >> 6.45 250.35 19.00 240087 0.00 0.00 > >> >> >> >> Curl_hash_add > >> >> >> >> 4.81 264.51 14.16 334257102 0.00 0.00 > >> >> >> >> Curl_str_key_compare > >> >> >> >> 3.40 274.52 10.01 119514 0.00 0.00 > create_conn > >> >> >> >> 0.93 277.25 2.73 4478910 0.00 0.00 > >> >> >> >> dprintf_formatf > >> >> >> >> 0.90 279.91 2.66 122209 0.00 0.00 > >> >> >> >> ConnectionStore > >> >> >> >> 0.40 281.09 1.18 > >> >> >> >> locking_function > >> >> >> >> 0.27 281.88 0.79 114759 0.00 0.00 > >> >> >> >> curl_multi_remove_handle > >> >> >> >> 0.21 282.51 0.63 7993664 0.00 0.00 > >> >> >> >> client_tracing_function > >> >> >> >> 0.21 283.12 0.61 122685 0.00 0.00 > >> >> >> >> Curl_num_addresses > >> >> >> >> 0.19 283.68 0.56 397028 0.00 0.00 > >> >> >> >> curl_multi_socket_action > >> >> >> >> 0.19 284.23 0.55 2036116 0.00 0.00 > >> >> >> >> multi_runsingle > >> >> >> >> 0.17 284.72 0.49 703687 0.00 0.00 > >> >> >> >> Curl_readwrite > >> >> >> >> 0.16 285.19 0.47 > id_function > >> >> >> >> 0.15 285.62 0.43 29692029 0.00 0.00 > >> >> >> >> Curl_socket_check > >> >> >> >> 0.14 286.02 0.40 91973856 0.00 0.00 > >> >> >> >> Curl_raw_toupper > >> >> >> >> 0.13 286.40 0.38 164664144 0.00 0.00 addbyter > >> >> >> >> 0.13 286.77 0.37 2180797 0.00 0.00 > Curl_infof > >> >> >> >> 0.11 287.10 0.33 122675 0.00 0.00 > >> >> >> >> Curl_if_is_interface_name > >> >> >> >> 0.10 287.39 0.29 471318 0.00 0.00 > >> >> >> >> ossl_connect_common > >> >> >> >> 0.10 287.67 0.28 2113528 0.00 0.00 > >> >> >> >> Curl_raw_equal > >> >> >> >> 0.09 287.93 0.26 8676 0.00 0.02 > >> >> >> >> curl_multi_socket_all > >> >> >> >> 0.08 288.17 0.24 1663262 0.00 0.00 > Curl_splay > >> >> >> >> 0.08 288.41 0.24 8505 0.00 0.01 > >> >> >> >> curl_multi_perform > >> >> >> >> 0.07 288.63 0.22 3114576 0.00 0.00 > >> >> >> >> Curl_pgrsUpdate > >> >> >> >> 0.06 288.80 0.17 119637 0.00 0.00 > >> >> >> >> singleipconnect > >> >> >> >> 0.06 288.97 0.17 7994818 0.00 0.00 > Curl_debug > >> >> >> >> 0.05 289.13 0.16 235820 0.00 0.00 event_del > >> >> >> >> 0.05 289.29 0.16 119393071 0.00 0.00 > >> >> >> >> curl_strequal > >> >> >> >> 0.05 289.45 0.16 18535473 0.00 0.00 > >> >> >> >> alloc_addbyter > >> >> >> >> 0.05 289.61 0.16 356074 0.00 0.00 > >> >> >> >> Curl_hash_str > >> >> >> >> 0.05 289.76 0.15 1797233 0.00 0.00 > >> >> >> >> fd_key_compare > >> >> >> >> 0.05 289.91 0.15 234966 0.00 0.00 > >> >> >> >> Curl_ssl_getsessionid > >> >> >> >> 0.04 290.04 0.13 585853 0.00 0.00 > >> >> >> >> Curl_llist_destroy > >> >> >> >> 0.04 290.16 0.12 240826 0.00 0.00 > Curl_resolv > >> >> >> >> 0.03 290.26 0.10 7514328 0.00 0.00 > >> >> >> >> Curl_raw_nequal > >> >> >> >> 0.03 290.36 0.10 2214436 0.00 0.00 Curl_read > >> >> >> >> 0.03 290.46 0.10 116872 0.00 0.00 > >> >> >> >> Curl_http_readwrite_headers > >> >> >> >> 0.03 290.56 0.10 > >> >> >> >> ssl_tls_trace > >> >> >> >> 0.03 290.66 0.10 8013097 0.00 0.00 > >> >> >> >> scan_response > >> >> >> >> 0.03 290.75 0.09 29488555 0.00 0.00 > >> >> >> >> curlx_tvdiff > >> >> >> >> 0.03 290.84 0.09 2142468 0.00 0.00 > Curl_setopt > >> >> >> >> 0.03 290.93 0.09 476299 0.00 0.00 > Curl_expire > >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> 2012/3/5 Robert Iakobashvili <cor...@gm...> > >> >> >> >>> > >> >> >> >>> Dear Fred Huang, > >> >> >> >>> > >> >> >> >>> Thanks for your posting. > >> >> >> >>> > >> >> >> >>> Could you please check if this is due to some badness at > cares? > >> >> >> >>> Thanks. > >> >> >> >>> > >> >> >> >>> Let's exclude usage of cares based DNS resolving by applying > the > >> >> >> >>> patch > >> >> >> >>> attached > >> >> >> >>> and use instead global DNS cache. > >> >> >> >>> > >> >> >> >>> cd ./curl-loader-0.56 > >> >> >> >>> patch -p1 < path-to/disable-cares-use-global-dns.patch > >> >> >> >>> make cleanall; make > >> >> >> >>> > >> >> >> >>> See if it helps. > >> >> >> >>> > >> >> >> >>> Global cache, unfortunately, is not thread safe. > >> >> >> >>> > >> >> >> >>> If it helps, we'll try to restore the previous version of > cares. > >> >> >> >>> > >> >> >> >>> Thanks, > >> >> >> >>> Robert > >> >> >> >>> > >> >> >> >>> On Mon, Mar 5, 2012 at 4:28 PM, Fred Huang <di...@gm...> > >> >> >> >>> wrote: > >> >> >> >>>> > >> >> >> >>>> test 1: > >> >> >> >>>> > >> >> >> >>>> dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, > >> >> >> >>>> resolve > >> >> >> >>>> *.com to one IP address > >> >> >> >>>> number of domain names in all https URLs: 780,000 > >> >> >> >>>> number of client: 1000 > >> >> >> >>>> CPU usage: 78% > >> >> >> >>>> cpu% irq% sirq% sys% iowt% mem_used buf&cached > >> >> >> >>>> 78.8 0.0 0.6 3.8 0.0 3640.9Mb 282.1Mb > >> >> >> >>>> SSL TPS: 300 > >> >> >> >>>> SSL throughput: 200Mbps > >> >> >> >>>> > >> >> >> >>>> # gprof /usr/bin/curl-loader gmon.out -p | head -50 > >> >> >> >>>> Flat profile: > >> >> >> >>>> Each sample counts as 0.01 seconds. > >> >> >> >>>> % cumulative self self total > >> >> >> >>>> time seconds seconds calls s/call s/call name > >> >> >> >>>> 38.55 127.10 127.10 130975 0.00 0.00 > >> >> >> >>>> Curl_hash_clean_with_criterium > >> >> >> >>>> 24.63 208.32 81.22 1247528407 0.00 0.00 > >> >> >> >>>> hostcache_timestamp_remove > >> >> >> >>>> 13.08 251.44 43.12 2791988 0.00 0.00 > >> >> >> >>>> Curl_hash_pick > >> >> >> >>>> 7.30 275.51 24.07 352490 0.00 0.00 > >> >> >> >>>> Curl_hash_add > >> >> >> >>>> 6.01 295.31 19.80 410760770 0.00 0.00 > >> >> >> >>>> Curl_str_key_compare > >> >> >> >>>> 2.95 305.04 9.73 131015 0.00 0.00 > >> >> >> >>>> create_conn > >> >> >> >>>> 0.88 307.94 2.90 4841390 0.00 0.00 > >> >> >> >>>> dprintf_formatf > >> >> >> >>>> 0.74 310.38 2.44 136036 0.00 0.00 > >> >> >> >>>> ConnectionStore > >> >> >> >>>> 0.48 311.96 1.58 814604911 0.00 0.00 > >> >> >> >>>> ares__is_list_empty > >> >> >> >>>> 0.36 313.16 1.20 > >> >> >> >>>> locking_function > >> >> >> >>>> 0.27 314.06 0.90 261608 0.00 0.00 > >> >> >> >>>> ares_cancel > >> >> >> >>>> 0.22 314.79 0.73 127417 0.00 0.00 > >> >> >> >>>> curl_multi_remove_handle > >> >> >> >>>> 0.19 315.40 0.62 8906301 0.00 0.00 > >> >> >> >>>> client_tracing_function > >> >> >> >>>> 0.17 315.97 0.57 135305 0.00 0.00 > >> >> >> >>>> Curl_num_addresses > >> >> >> >>>> 0.17 316.54 0.57 181937907 0.00 0.00 > addbyter > >> >> >> >>>> 0.17 317.09 0.55 2374363 0.00 0.00 > >> >> >> >>>> multi_runsingle > >> >> >> >>>> 0.16 317.62 0.53 541954 0.00 0.00 > >> >> >> >>>> ares__init_list_node > >> >> >> >>>> 0.16 318.14 0.52 494412 0.00 0.00 > >> >> >> >>>> curl_multi_socket_action > >> >> >> >>>> 0.14 318.60 0.46 33578897 0.00 0.00 > >> >> >> >>>> Curl_socket_check > >> >> >> >>>> 0.13 319.04 0.44 102951052 0.00 0.00 > >> >> >> >>>> Curl_raw_toupper > >> >> >> >>>> 0.13 319.48 0.44 787558 0.00 0.00 > >> >> >> >>>> Curl_readwrite > >> >> >> >>>> 0.13 319.91 0.44 > >> >> >> >>>> id_function > >> >> >> >>>> 0.12 320.29 0.38 391390 0.00 0.00 > >> >> >> >>>> Curl_hash_str > >> >> >> >>>> 0.09 320.58 0.29 9216 0.00 0.00 > >> >> >> >>>> curl_multi_perform > >> >> >> >>>> 0.08 320.86 0.28 2350908 0.00 0.00 > >> >> >> >>>> Curl_raw_equal > >> >> >> >>>> 0.08 321.14 0.28 2019415 0.00 0.00 > >> >> >> >>>> Curl_splay > >> >> >> >>>> 0.08 321.40 0.26 520625 0.00 0.00 > >> >> >> >>>> ossl_connect_common > >> >> >> >>>> 0.08 321.65 0.25 3532992 0.00 0.00 > >> >> >> >>>> Curl_pgrsUpdate > >> >> >> >>>> 0.07 321.87 0.22 131778537 0.00 0.00 > >> >> >> >>>> curl_strequal > >> >> >> >>>> 0.05 322.05 0.18 8954118 0.00 0.00 > >> >> >> >>>> scan_response > >> >> >> >>>> 0.05 322.23 0.18 533644 0.00 0.00 > >> >> >> >>>> ares_expand_name > >> >> >> >>>> 0.05 322.40 0.17 2477345 0.00 0.00 > >> >> >> >>>> fd_key_compare > >> >> >> >>>> 0.05 322.57 0.17 2429135 0.00 0.00 > >> >> >> >>>> Curl_infof > >> >> >> >>>> 0.05 322.74 0.17 130966 0.00 0.00 > >> >> >> >>>> singleipconnect > >> >> >> >>>> 0.05 322.90 0.16 9292 0.00 0.00 > >> >> >> >>>> curl_multi_socket_all > >> >> >> >>>> 0.05 323.06 0.16 270137 0.00 0.00 > >> >> >> >>>> ares__get_hostent > >> >> >> >>>> 0.05 323.21 0.15 8943657 0.00 0.00 > >> >> >> >>>> Curl_debug > >> >> >> >>>> 0.04 323.35 0.14 262493 0.00 0.00 > >> >> >> >>>> Curl_ssl_getsessionid > >> >> >> >>>> 0.04 323.48 0.13 566006 0.00 0.00 > >> >> >> >>>> socket_callback > >> >> >> >>>> 0.04 323.61 0.13 33983864 0.00 0.00 > >> >> >> >>>> curlx_tvdiff > >> >> >> >>>> 0.04 323.74 0.13 132853 0.00 0.00 > >> >> >> >>>> Curl_http_readwrite_headers > >> >> >> >>>> 0.04 323.86 0.12 344032 0.00 0.00 > epoll_del > >> >> >> >>>> 0.04 323.98 0.12 223477 0.00 0.00 > Curl_poll > >> >> >> >>>> 0.04 324.10 0.12 8526280 0.00 0.00 > >> >> >> >>>> Curl_raw_nequal > >> >> >> >>>> 0.03 324.21 0.11 348839 0.00 0.00 > event_del > >> >> >> >>>> > >> >> >> >>>> > >> >> >> >>>> > >> >> >> >>>> > >> >> >> >>>> test 2: > >> >> >> >>>> > >> >> >> >>>> dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, > >> >> >> >>>> resolve > >> >> >> >>>> *.com to one IP address > >> >> >> >>>> number of domain names in all https URLs: 1 > >> >> >> >>>> number of client: 1000 > >> >> >> >>>> CPU usage: 75% > >> >> >> >>>> SSL TPS: 1300 > >> >> >> >>>> SSL throughput: 700Mbps > >> >> >> >>>> > >> >> >> >>>> # gprof /usr/bin/curl-loader gmon.out -p | head -50 > >> >> >> >>>> Flat profile: > >> >> >> >>>> Each sample counts as 0.01 seconds. > >> >> >> >>>> % cumulative self self total > >> >> >> >>>> time seconds seconds calls s/call s/call name > >> >> >> >>>> 8.98 2.45 2.45 5127312 0.00 0.00 > >> >> >> >>>> dprintf_formatf > >> >> >> >>>> 8.90 4.88 2.43 1292796503 0.00 0.00 > >> >> >> >>>> ares__is_list_empty > >> >> >> >>>> 6.12 6.55 1.67 322461 0.00 0.00 > >> >> >> >>>> create_conn > >> >> >> >>>> 4.58 7.80 1.25 419872 0.00 0.00 > >> >> >> >>>> ares_cancel > >> >> >> >>>> 4.10 8.92 1.12 2100162 0.00 0.00 > >> >> >> >>>> Curl_readwrite > >> >> >> >>>> 3.92 9.99 1.07 4527388 0.00 0.00 > >> >> >> >>>> Curl_hash_pick > >> >> >> >>>> 3.43 10.93 0.94 > >> >> >> >>>> locking_function > >> >> >> >>>> 3.04 11.76 0.83 909759 0.00 0.00 > >> >> >> >>>> curl_multi_socket_action > >> >> >> >>>> 3.00 12.58 0.82 97252 0.00 0.00 > >> >> >> >>>> Curl_hash_clean_with_criterium > >> >> >> >>>> 2.71 13.32 0.74 179704016 0.00 0.00 > >> >> >> >>>> Curl_raw_toupper > >> >> >> >>>> 2.68 14.05 0.73 4646524 0.00 0.00 > >> >> >> >>>> multi_runsingle > >> >> >> >>>> 2.44 14.71 0.67 13862992 0.00 0.00 > >> >> >> >>>> client_tracing_function > >> >> >> >>>> 2.42 15.37 0.66 319735 0.00 0.00 > >> >> >> >>>> curl_multi_remove_handle > >> >> >> >>>> 2.42 16.03 0.66 22 0.03 0.03 > >> >> >> >>>> ares__init_list_node > >> >> >> >>>> 1.94 16.56 0.53 9930244 0.00 0.00 > >> >> >> >>>> hostcache_timestamp_remove > >> >> >> >>>> 1.94 17.09 0.53 169320088 0.00 0.00 > addbyter > >> >> >> >>>> 1.80 17.58 0.49 > >> >> >> >>>> id_function > >> >> >> >>>> 1.36 17.95 0.37 7057146 0.00 0.00 > >> >> >> >>>> Curl_pgrsUpdate > >> >> >> >>>> 1.36 18.32 0.37 16835 0.00 0.00 > >> >> >> >>>> curl_multi_socket_all > >> >> >> >>>> 1.25 18.66 0.34 16719 0.00 0.00 > >> >> >> >>>> curl_multi_perform > >> >> >> >>>> 1.21 18.99 0.33 329403 0.00 0.00 > >> >> >> >>>> Curl_http_readwrite_headers > >> >> >> >>>> 1.17 19.31 0.32 4036624 0.00 0.00 > >> >> >> >>>> Curl_splay > >> >> >> >>>> 1.14 19.62 0.31 20838902 0.00 0.00 > >> >> >> >>>> Curl_raw_nequal > >> >> >> >>>> 0.82 19.85 0.23 2764123 0.00 0.00 > >> >> >> >>>> Curl_infof > >> >> >> >>>> 0.81 20.07 0.22 5933857 0.00 0.00 > ossl_recv > >> >> >> >>>> 0.77 20.28 0.21 323112 0.00 0.00 > >> >> >> >>>> Curl_splayremovebyaddr > >> >> >> >>>> 0.75 20.48 0.21 5943991 0.00 0.00 > Curl_read > >> >> >> >>>> 0.70 20.67 0.19 425174 0.00 0.00 > event_del > >> >> >> >>>> 0.70 20.86 0.19 98652 0.00 0.00 > >> >> >> >>>> Curl_if_is_interface_name > >> >> >> >>>> 0.70 21.05 0.19 8677733 0.00 0.00 > >> >> >> >>>> Curl_socket_check > >> >> >> >>>> 0.59 21.21 0.16 4129554 0.00 0.00 > >> >> >> >>>> fd_key_compare > >> >> >> >>>> 0.57 21.37 0.16 14017019 0.00 0.00 > >> >> >> >>>> Curl_debug > >> >> >> >>>> 0.55 21.52 0.15 9376861 0.00 0.00 > >> >> >> >>>> stat_data_in_add > >> >> >> >>>> 0.55 21.67 0.15 6840468 0.00 0.00 > >> >> >> >>>> Curl_timeleft > >> >> >> >>>> 0.55 21.82 0.15 2054602 0.00 0.00 > >> >> >> >>>> Curl_raw_equal > >> >> >> >>>> 0.55 21.97 0.15 1291611 0.00 0.00 > >> >> >> >>>> Curl_expire > >> >> >> >>>> 0.51 22.11 0.14 16570063 0.00 0.00 > >> >> >> >>>> curlx_tvnow > >> >> >> >>>> 0.51 22.25 0.14 13863284 0.00 0.00 > >> >> >> >>>> scan_response > >> >> >> >>>> 0.51 22.39 0.14 5498864 0.00 0.00 > >> >> >> >>>> Curl_setopt > >> >> >> >>>> 0.48 22.52 0.13 97490291 0.00 0.00 > >> >> >> >>>> curl_strequal > >> >> >> >>>> 0.44 22.64 0.12 324253 0.00 0.00 > Curl_http > >> >> >> >>>> 0.44 22.76 0.12 394386 0.00 0.00 > >> >> >> >>>> ossl_connect_common > >> >> >> >>>> 0.40 22.87 0.11 3581335 0.00 0.00 > >> >> >> >>>> Curl_getinfo > >> >> >> >>>> 0.38 22.97 0.11 26983092 0.00 0.00 > >> >> >> >>>> alloc_addbyter > >> >> >> >>>> 0.37 23.07 0.10 8729044 0.00 0.00 > >> >> >> >>>> Curl_client_write > >> >> >> >>>> > >> >> >> >>>> > >> >> >> >>>> > >> >> >> >>>> > >> >> >> >>>> > >> >> >> >>>> > ------------------------------------------------------------------------------ > >> >> >> >>>> Try before you buy = See our experts in action! > >> >> >> >>>> The most comprehensive online learning library for Microsoft > >> >> >> >>>> developers > >> >> >> >>>> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, > >> >> >> >>>> CSS3, > >> >> >> >>>> MVC3, > >> >> >> >>>> Metro Style Apps, more. Free future releases when you > subscribe > >> >> >> >>>> now! > >> >> >> >>>> http://p.sf.net/sfu/learndevnow-dev2 > >> >> >> >>>> _______________________________________________ > >> >> >> >>>> curl-loader-devel mailing list > >> >> >> >>>> cur...@li... > >> >> >> >>>> > https://lists.sourceforge.net/lists/listinfo/curl-loader-devel > >> >> >> >>>> > >> >> >> >>> > >> >> >> >>> > >> >> >> >>> > >> >> >> >>> -- > >> >> >> >>> Regards, > >> >> >> >>> Robert Iakobashvili, Ph.D. > >> >> >> >>> > >> >> >> >>> Home: http://www.ghotit.com > >> >> >> >>> Blog: http://www.ghotit.com/category/ghotit-blogs/ > >> >> >> >>> Twitter: http://twitter.com/ghotit > >> >> >> >>> Facebook: http://facebook.com/ghotit > >> >> >> >>> > >> >> >> >>> > >> >> >> >>> > ...................................................................... > >> >> >> >>> Ghotit Dyslexia > >> >> >> >>> Assistive technology that understands you > >> >> >> >>> > >> >> >> >>> > >> >> >> >>> > ...................................................................... > >> >> >> >>> > >> >> >> >>> > >> >> >> >>> > >> >> >> >>> > >> >> >> >>> > >> >> >> >>> > ------------------------------------------------------------------------------ > >> >> >> >>> Try before you buy = See our experts in action! > >> >> >> >>> The most comprehensive online learning library for Microsoft > >> >> >> >>> developers > >> >> >> >>> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, > >> >> >> >>> CSS3, > >> >> >> >>> MVC3, > >> >> >> >>> Metro Style Apps, more. Free future releases when you > subscribe > >> >> >> >>> now! > >> >> >> >>> http://p.sf.net/sfu/learndevnow-dev2 > >> >> >> >>> _______________________________________________ > >> >> >> >>> curl-loader-devel mailing list > >> >> >> >>> cur...@li... > >> >> >> >>> > https://lists.sourceforge.net/lists/listinfo/curl-loader-devel > >> >> >> >>> > >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> > ------------------------------------------------------------------------------ > >> >> >> >> Try before you buy = See our experts in action! > >> >> >> >> The most comprehensive online learning library for Microsoft > >> >> >> >> developers > >> >> >> >> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, > >> >> >> >> CSS3, > >> >> >> >> MVC3, > >> >> >> >> Metro Style Apps, more. Free future releases when you subscribe > >> >> >> >> now! > >> >> >> >> http://p.sf.net/sfu/learndevnow-dev2 > >> >> >> >> _______________________________________________ > >> >> >> >> curl-loader-devel mailing list > >> >> >> >> cur...@li... > >> >> >> >> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel > >> >> >> >> > >> >> >> > > >> >> >> > > >> >> >> > > >> >> >> > -- > >> >> >> > Regards, > >> >> >> > Robert Iakobashvili, Ph.D. > >> >> >> > > >> >> >> > Home: http://www.ghotit.com > >> >> >> > Blog: http://www.ghotit.com/category/ghotit-blogs/ > >> >> >> > Twitter: http://twitter.com/ghotit > >> >> >> > Facebook: http://facebook.com/ghotit > >> >> >> > > >> >> >> > > >> >> >> > > ...................................................................... > >> >> >> > Ghotit Dyslexia > >> >> >> > Assistive technology that understands you > >> >> >> > > >> >> >> > > >> >> >> > > ...................................................................... > >> >> >> > >> >> >> > >> >> >> > >> >> >> -- > >> >> >> Regards, > >> >> >> Robert Iakobashvili, Ph.D. > >> >> >> > >> >> >> Home: http://www.ghotit.com > >> >> >> Blog: http://www.ghotit.com/category/ghotit-blogs/ > >> >> >> Twitter: http://twitter.com/ghotit > >> >> >> Facebook: http://facebook.com/ghotit > >> >> >> > >> >> >> > ...................................................................... > >> >> >> Ghotit Dyslexia > >> >> >> Assistive technology that understands you > >> >> >> > >> >> >> > ...................................................................... > >> >> >> > >> >> >> > >> >> >> > >> >> >> > >> >> >> > ------------------------------------------------------------------------------ > >> >> >> Try before you buy = See our experts in action! > >> >> >> The most comprehensive online learning library for Microsoft > >> >> >> developers > >> >> >> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, > >> >> >> MVC3, > >> >> >> Metro Style Apps, more. Free future releases when you subscribe > now! > >> >> >> http://p.sf.net/sfu/learndevnow-dev2 > >> >> >> _______________________________________________ > >> >> >> curl-loader-devel mailing list > >> >> >> cur...@li... > >> >> >> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel > >> >> > > >> >> > > >> >> > > >> >> > > >> >> > > >> >> > > ------------------------------------------------------------------------------ > >> >> > Keep Your Developer Skills Current with LearnDevNow! > >> >> > The most comprehensive online learning library for Microsoft > >> >> > developers > >> >> > is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, > >> >> > MVC3, > >> >> > Metro Style Apps, more. Free future releases when you subscribe > now! > >> >> > http://p.sf.net/sfu/learndevnow-d2d > >> >> > _______________________________________________ > >> >> > curl-loader-devel mailing list > >> >> > cur...@li... > >> >> > https://lists.sourceforge.net/lists/listinfo/curl-loader-devel > >> >> > > >> >> > >> >> > >> >> > >> >> -- > >> >> Regards, > >> >> Robert Iakobashvili, Ph.D. > >> >> > >> >> Home: http://www.ghotit.com > >> >> Blog: http://www.ghotit.com/category/ghotit-blogs/ > >> >> Twitter: http://twitter.com/ghotit > >> >> Facebook: http://facebook.com/ghotit > >> >> > ...................................................................... > >> >> Ghotit Dyslexia > >> >> Assistive technology that understands you > >> >> > ...................................................................... > >> >> > >> >> > >> >> > >> >> > ------------------------------------------------------------------------------ > >> >> Keep Your Developer Skills Current with LearnDevNow! > >> >> The most comprehensive online learning library for Microsoft > developers > >> >> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, > >> >> MVC3, > >> >> Metro Style Apps, more. Free future releases when you subscribe now! > >> >> http://p.sf.net/sfu/learndevnow-d2d > >> >> _______________________________________________ > >> >> curl-loader-devel mailing list > >> >> cur...@li... > >> >> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel > >> > > >> > > >> > > >> > > >> > > ------------------------------------------------------------------------------ > >> > Keep Your Developer Skills Current with LearnDevNow! > >> > The most comprehensive online learning library for Microsoft > developers > >> > is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, > MVC3, > >> > Metro Style Apps, more. Free future releases when you subscribe now! > >> > http://p.sf.net/sfu/learndevnow-d2d > >> > _______________________________________________ > >> > curl-loader-devel mailing list > >> > cur...@li... > >> > https://lists.sourceforge.net/lists/listinfo/curl-loader-devel > >> > > >> > >> > >> > >> -- > >> Regards, > >> Robert Iakobashvili, Ph.D. > >> > >> Home: http://www.ghotit.com > >> Blog: http://www.ghotit.com/category/ghotit-blogs/ > >> Twitter: http://twitter.com/ghotit > >> Facebook: http://facebook.com/ghotit > >> ...................................................................... > >> Ghotit Dyslexia > >> Assistive technology that understands you > >> ...................................................................... > >> > >> > >> > ------------------------------------------------------------------------------ > >> Keep Your Developer Skills Current with LearnDevNow! > >> The most comprehensive online learning library for Microsoft developers > >> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, > >> Metro Style Apps, more. Free future releases when you subscribe now! > >> http://p.sf.net/sfu/learndevnow-d2d > >> _______________________________________________ > >> curl-loader-devel mailing list > >> cur...@li... > >> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel > > > > > > > > > ------------------------------------------------------------------------------ > > Virtualization & Cloud Management Using Capacity Planning > > Cloud computing makes use of virtualization - but cloud computing > > also focuses on allowing computing to be delivered as a service. > > http://www.accelacomm.com/jaw/sfnl/114/51521223/ > > _______________________________________________ > > curl-loader-devel mailing list > > cur...@li... > > https://lists.sourceforge.net/lists/listinfo/curl-loader-devel > > > > > > -- > Regards, > Robert Iakobashvili, Ph.D. > > Home: http://www.ghotit.com > Blog: http://www.ghotit.com/category/ghotit-blogs/ > Twitter: http://twitter.com/ghotit > Facebook: http://facebook.com/ghotit > ...................................................................... > Ghotit Dyslexia > Assistive technology that understands you > ...................................................................... > > > ------------------------------------------------------------------------------ > Virtualization & Cloud Management Using Capacity Planning > Cloud computing makes use of virtualization - but cloud computing > also focuses on allowing computing to be delivered as a service. > http://www.accelacomm.com/jaw/sfnl/114/51521223/ > _______________________________________________ > curl-loader-devel mailing list > cur...@li... > https://lists.sourceforge.net/lists/listinfo/curl-loader-devel > |
|
From: Robert I. <cor...@gm...> - 2012-03-07 06:38:08
|
Fred, On more suggestion, which is a general one. If N is the number of your logical CPUs, usually, I'd recommend to run 2*N + B threads. Thus, you can try: -t 16 -t 32 and it MAY increased a bit CAPS. Thanks, Robert On Wed, Mar 7, 2012 at 2:43 AM, Fred Huang <di...@gm...> wrote: > yes, I use -t 8 option because the test box has 8 logic CPUs (Dell 990, > Intel Core i7 2600, 16GB memory). > if I do not use -t option, the CPU used by curl-loader will be 100% on one > of 8 logic CPU (12% of total CPU), and the CAPS drops to 100: > # gprof /usr/bin/curl-loader gmon.out -p | head -20 > Flat profile: > Each sample counts as 0.01 seconds. > % cumulative self self total > time seconds seconds calls s/call s/call name > 34.12 2.74 2.74 17385 0.00 0.00 create_conn > 11.21 3.64 0.90 30998583 0.00 0.00 > hostcache_timestamp_remove > 9.84 4.43 0.79 13433 0.00 0.00 > Curl_hash_clean_with_criterium > 7.97 5.07 0.64 381818 0.00 0.00 Curl_hash_pick > 5.85 5.54 0.47 17385 0.00 0.00 ConnectionStore > 5.11 5.95 0.41 16385 0.00 0.00 > curl_multi_remove_handle > 4.11 6.28 0.33 16509889 0.00 0.00 Curl_str_key_compare > 3.36 6.55 0.27 23959914 0.00 0.00 Curl_socket_check > 3.24 6.81 0.26 56703 0.00 0.00 Curl_hash_add > 2.62 7.02 0.21 635750 0.00 0.00 dprintf_formatf > 1.25 7.12 0.10 92421190 0.00 0.00 ares__is_list_empty > 0.75 7.18 0.06 97957 0.00 0.00 Curl_readwrite > 0.62 7.23 0.05 23422260 0.00 0.00 addbyter > 0.62 7.28 0.05 1136775 0.00 0.00 > client_tracing_function > 0.50 7.32 0.04 3091375 0.00 0.00 ares__init_list_head > > I also tried to use > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, 20); > but get no improvement on CAPS. > > > Regards, > Fred > > 2012/3/7 Robert Iakobashvili <cor...@gm...> >> >> Dear Fred, >> >> If I understand correctly your observations, the issue is not in: >> >> 1. DNS resolving; >> 2. Version of cares or libcurl; >> >> but rather in >> >> Less effective handling of HTTPS versus HTTP in libcurl. >> >> Please, confirm that my understanding is correct. >> >> >> HTTPS URLs are using an extra locking in openssl with -t option >> that could be making the issue. >> If you are using -t option, try to load without it and let's say with >> 300 clients. >> >> If you are not using -t, thus the only way to proceed is to share >> your observations with libcurl people at >> >> libcurl development <cur...@co...> >> >> Thank you. >> >> Best wishes. >> Robert >> >> On Tue, Mar 6, 2012 at 6:17 PM, Fred Huang <di...@gm...> wrote: >> > 1. the major CPU consumer is curl-loader. here is the top: >> > top - 07:36:36 up 23:31, 5 users, load average: 4.32, 2.26, 0.91 >> > Tasks: 237 total, 2 running, 235 sleeping, 0 stopped, 0 zombie >> > Cpu0 : 76.6%us, 3.6%sy, 0.0%ni, 19.4%id, 0.0%wa, 0.0%hi, 0.3%si, >> > 0.0% >> > Cpu1 : 72.4%us, 4.0%sy, 0.0%ni, 22.9%id, 0.0%wa, 0.0%hi, 0.7%si, >> > 0.0% >> > Cpu2 : 65.8%us, 3.3%sy, 0.0%ni, 30.6%id, 0.0%wa, 0.0%hi, 0.3%si, >> > 0.0% >> > Cpu3 : 75.2%us, 4.3%sy, 0.0%ni, 19.5%id, 0.0%wa, 0.0%hi, 1.0%si, >> > 0.0% >> > Cpu4 : 79.1%us, 4.0%sy, 0.0%ni, 16.8%id, 0.0%wa, 0.0%hi, 0.0%si, >> > 0.0% >> > Cpu5 : 75.8%us, 4.3%sy, 0.0%ni, 19.2%id, 0.0%wa, 0.0%hi, 0.7%si, >> > 0.0% >> > Cpu6 : 74.1%us, 5.0%sy, 0.0%ni, 20.3%id, 0.0%wa, 0.0%hi, 0.7%si, >> > 0.0% >> > Cpu7 : 78.7%us, 4.0%sy, 0.0%ni, 16.6%id, 0.0%wa, 0.0%hi, 0.7%si, >> > 0.0% >> > Mem: 16209932k total, 3829500k used, 12380432k free, 99112k buffers >> > Swap: 18448376k total, 0k used, 18448376k free, 320284k cached >> > >> > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND >> > 30795 root 20 0 3200m 2.8g 2296 S 79.2 18.4 21:26.84 curl-loader >> > 1832 nobody 20 0 13944 1736 580 S 0.1 0.0 0:32.65 dnsmasq >> > 9 root 20 0 0 0 0 S 0.0 0.0 0:00.53 ksoftirqd/1 >> > 30825 root 20 0 15220 1360 960 R 0.0 0.0 0:00.42 top >> > 1 root 20 0 19400 1568 1256 S 0.0 0.0 0:00.75 init >> > 2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd >> > 3 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/0 >> > 4 root 20 0 0 0 0 S 0.0 0.0 0:00.06 ksoftirqd/0 >> > 5 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/0 >> > >> > >> > 2. 1000 loading clients >> > >> > 3. yes, tried 0.52 but no lucky -- CAPS 380, throughput 200Mbps, CPU 90% >> > top - 08:17:10 up 1 day, 12 min, 8 users, load average: 4.66, 3.39, >> > 1.88 >> > Tasks: 245 total, 2 running, 243 sleeping, 0 stopped, 0 zombie >> > Cpu0 : 82.2%us, 4.0%sy, 0.0%ni, 13.9%id, 0.0%wa, 0.0%hi, 0.0%si, >> > 0.0% >> > Cpu1 : 86.1%us, 4.3%sy, 0.0%ni, 7.9%id, 0.0%wa, 0.0%hi, 1.7%si, >> > 0.0% >> > Cpu2 : 85.3%us, 4.0%sy, 0.0%ni, 10.0%id, 0.0%wa, 0.0%hi, 0.7%si, >> > 0.0% >> > Cpu3 : 81.8%us, 3.0%sy, 0.0%ni, 14.2%id, 0.0%wa, 0.0%hi, 1.0%si, >> > 0.0% >> > Cpu4 : 83.7%us, 4.0%sy, 0.0%ni, 12.3%id, 0.0%wa, 0.0%hi, 0.0%si, >> > 0.0% >> > Cpu5 : 92.7%us, 3.3%sy, 0.0%ni, 3.3%id, 0.0%wa, 0.0%hi, 0.7%si, >> > 0.0% >> > Cpu6 : 89.7%us, 2.0%sy, 0.0%ni, 8.0%id, 0.0%wa, 0.0%hi, 0.3%si, >> > 0.0% >> > Cpu7 : 88.1%us, 3.6%sy, 0.0%ni, 6.6%id, 0.0%wa, 0.0%hi, 1.7%si, >> > 0.0% >> > Mem: 16209932k total, 1795748k used, 14414184k free, 102824k buffers >> > Swap: 18448376k total, 0k used, 18448376k free, 382988k cached >> > >> > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND >> > 12236 root 20 0 1150m 894m 2572 S 90.7 5.7 51:11.91 curl-loader >> > 1832 nobody 20 0 13944 1736 580 S 0.1 0.0 0:47.04 dnsmasq >> > >> > ]# gprof /usr/bin/curl-loader gmon.out -p | head -20 >> > Flat profile: >> > Each sample counts as 0.01 seconds. >> > % cumulative self self total >> > time seconds seconds calls s/call s/call name >> > 43.92 151.95 151.95 157295 0.00 0.00 >> > Curl_hash_clean_with_criterium >> > 22.90 231.17 79.22 1712190971 0.00 0.00 >> > hostcache_timestamp_remove >> > 13.15 276.68 45.51 3260321 0.00 0.00 Curl_hash_pick >> > 7.51 302.65 25.97 394171 0.00 0.00 Curl_hash_add >> > 5.98 323.34 20.68 567437518 0.00 0.00 >> > Curl_str_key_compare >> > 1.00 326.80 3.46 5380220 0.00 0.00 dprintf_formatf >> > 0.41 328.21 1.41 locking_function >> > 0.32 329.32 1.11 469222138 0.00 0.00 >> > ares__is_list_empty >> > 0.31 330.40 1.08 3284508 0.00 0.00 multi_runsingle >> > 0.28 331.38 0.98 10006806 0.00 0.00 >> > client_tracing_function >> > 0.24 332.21 0.83 151951 0.00 0.00 Curl_num_addresses >> > 0.22 332.97 0.77 188075672 0.00 0.00 addbyter >> > 0.19 333.64 0.67 841840 0.00 0.00 Curl_readwrite >> > 0.19 334.29 0.65 id_function >> > 0.18 334.90 0.61 39843 0.00 0.01 >> > curl_multi_socket_all >> > >> > >> > >> > 4. yes, it is correct for 0.56. If use same URL list by replacing https >> > to >> > http, I can get CAPS 1500 and throughput 870Mbps with CPU% < 60% >> > >> > >> > 2012/3/6 Robert Iakobashvili <cor...@gm...> >> >> >> >> Several questions: >> >> >> >> 1. Could you please run top and see if there are other processes >> >> burning >> >> CPU? >> >> 2. What is the number of loading clients that you use? >> >> 3. Could you try curl-loader-0.52 if it compiles for you? >> >> If not, please try to place curl-loader-7.19.7.tar.gz and >> >> c-ares-16.0.tar.gz, >> >> update their versions in Makefile and rebuild by make cleanall; make >> >> >> >> 4. Is it correct to say that for HTTP URLs (not HTTPS) you are getting >> >> a good throughput >> >> and CAPS with curl-loader-0.56? >> >> >> >> Thanks, >> >> Robert >> >> >> >> On Tue, Mar 6, 2012 at 3:52 AM, Fred Huang <di...@gm...> wrote: >> >> > TIMEOUT -1 -> 60 works a little better but no performance improvement >> >> > on >> >> > CAPS or throughput. >> >> > >> >> > test 5: >> >> > enable c-ares; >> >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, 60); >> >> > >> >> > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve >> >> > *.com >> >> > to >> >> > one IP address >> >> > number of domain names in URLs: 780,000 >> >> > number of client: 1000 >> >> > CPU usage: 74% >> >> > cpu% irq% sirq% sys% iowt% mem_used buf&cached >> >> > 74.0 0.0 0.7 4.2 0.0 3530.6Mb 250.8Mb >> >> > SSL TPS: 360 >> >> > SSL throughput: 220Mbps >> >> > >> >> > # gprof /usr/bin/curl-loader gmon.out -p | head -50 >> >> > Flat profile: >> >> > Each sample counts as 0.01 seconds. >> >> > % cumulative self self total >> >> > time seconds seconds calls s/call s/call name >> >> > 26.63 23.13 23.13 119226 0.00 0.00 >> >> > Curl_hash_clean_with_criterium >> >> > 20.34 40.80 17.67 317792146 0.00 0.00 >> >> > hostcache_timestamp_remove >> >> > 11.27 50.59 9.79 2579903 0.00 0.00 Curl_hash_pick >> >> > 7.84 57.40 6.81 120606 0.00 0.00 create_conn >> >> > 5.65 62.31 4.91 119117401 0.00 0.00 >> >> > Curl_str_key_compare >> >> > 4.98 66.64 4.33 329829 0.00 0.00 Curl_hash_add >> >> > 2.80 69.08 2.44 4476606 0.00 0.00 dprintf_formatf >> >> > 2.57 71.31 2.23 124019 0.00 0.00 ConnectionStore >> >> > 1.50 72.61 1.30 740997609 0.00 0.00 >> >> > ares__is_list_empty >> >> > 1.16 73.61 1.01 >> >> > locking_function >> >> > 0.76 74.27 0.66 238744 0.00 0.00 ares_cancel >> >> > 0.70 74.88 0.61 457564 0.00 0.00 >> >> > curl_multi_socket_action >> >> > 0.67 75.46 0.58 118616 0.00 0.00 >> >> > curl_multi_remove_handle >> >> > 0.58 75.97 0.51 8211502 0.00 0.00 >> >> > client_tracing_function >> >> > 0.53 76.43 0.46 733561 0.00 0.00 Curl_readwrite >> >> > 0.51 76.87 0.44 2183179 0.00 0.00 multi_runsingle >> >> > 0.51 77.31 0.44 166585567 0.00 0.00 addbyter >> >> > 0.44 77.69 0.38 2175805 0.00 0.00 Curl_raw_equal >> >> > 0.43 78.06 0.38 id_function >> >> > 0.40 78.41 0.35 29789346 0.00 0.00 >> >> > Curl_socket_check >> >> > 0.40 78.76 0.35 479671 0.00 0.00 >> >> > ossl_connect_common >> >> > 0.40 79.11 0.35 490639 0.00 0.00 >> >> > ares__init_list_node >> >> > 0.33 79.40 0.29 94197670 0.00 0.00 >> >> > Curl_raw_toupper >> >> > 0.33 79.69 0.29 3268031 0.00 0.00 Curl_pgrsUpdate >> >> > 0.28 79.93 0.24 8797 0.00 0.00 >> >> > curl_multi_perform >> >> > 0.27 80.17 0.24 2233641 0.00 0.00 Curl_infof >> >> > 0.26 80.40 0.23 1839837 0.00 0.00 Curl_splay >> >> > 0.24 80.61 0.21 121713028 0.00 0.00 curl_strequal >> >> > 0.21 80.79 0.18 243333 0.00 0.00 >> >> > Curl_ssl_getsessionid >> >> > 0.18 80.95 0.16 122180 0.00 0.00 >> >> > Curl_num_addresses >> >> > 0.17 81.09 0.15 8271666 0.00 0.00 Curl_debug >> >> > 0.16 81.23 0.14 18902452 0.00 0.00 alloc_addbyter >> >> > 0.16 81.37 0.14 8851 0.00 0.00 >> >> > curl_multi_socket_all >> >> > 0.14 81.50 0.13 2236769 0.00 0.00 Curl_read >> >> > 0.14 81.62 0.12 2271700 0.00 0.00 fd_key_compare >> >> > 0.14 81.74 0.12 323167 0.00 0.00 event_del >> >> > 0.14 81.86 0.12 205436 0.00 0.00 >> >> > read_udp_packets >> >> > 0.14 81.98 0.12 200979 0.00 0.00 processfds >> >> > 0.13 82.09 0.11 123886 0.00 0.00 >> >> > Curl_if_is_interface_name >> >> > 0.13 82.20 0.11 120350 0.00 0.00 Curl_http >> >> > 0.13 82.31 0.11 ssl_tls_trace >> >> > 0.12 82.41 0.10 119229 0.00 0.00 >> >> > Curl_splayremovebyaddr >> >> > 0.12 82.51 0.10 7798717 0.00 0.00 Curl_raw_nequal >> >> > 0.12 82.61 0.10 784485 0.00 0.00 event_add >> >> > 0.10 82.70 0.09 643482 0.00 0.00 Curl_expire >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > test 6: >> >> > enable c-ares; >> >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, 60); >> >> > CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.23.0 >> >> > >> >> > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve >> >> > *.com >> >> > to >> >> > one IP address >> >> > number of domain names in URLs: 780,000 >> >> > number of client: 1000 >> >> > CPU usage: 86% >> >> > cpu% irq% sirq% sys% iowt% mem_used buf&cached >> >> > 85.9 0.0 0.2 2.1 0.0 3537.2Mb 261.4Mb >> >> > SSL TPS: 360 >> >> > SSL throughput: 200Mbps >> >> > >> >> > # gprof /usr/bin/curl-loader gmon.out -p | head -20 >> >> > Flat profile: >> >> > Each sample counts as 0.01 seconds. >> >> > % cumulative self self total >> >> > time seconds seconds calls s/call s/call name >> >> > 27.19 19.96 19.96 103472 0.00 0.00 >> >> > Curl_hash_clean_with_criterium >> >> > 21.73 35.91 15.95 274168981 0.00 0.00 >> >> > hostcache_timestamp_remove >> >> > 10.33 43.49 7.58 2256441 0.00 0.00 Curl_hash_pick >> >> > 7.21 48.78 5.29 104540 0.00 0.00 create_conn >> >> > 6.20 53.33 4.55 103413405 0.00 0.00 >> >> > Curl_str_key_compare >> >> > 4.88 56.91 3.58 287109 0.00 0.00 Curl_hash_add >> >> > 3.02 59.13 2.22 3797855 0.00 0.00 dprintf_formatf >> >> > 2.10 60.67 1.54 107711 0.00 0.00 ConnectionStore >> >> > 1.65 61.88 1.21 642725327 0.00 0.00 >> >> > ares__is_list_empty >> >> > 1.19 62.76 0.88 >> >> > locking_function >> >> > 0.87 63.40 0.64 207817 0.00 0.00 ares_cancel >> >> > 0.56 63.81 0.41 404108 0.00 0.00 >> >> > curl_multi_socket_action >> >> > 0.54 64.21 0.40 425603 0.00 0.00 >> >> > ares__init_list_node >> >> > 0.53 64.60 0.39 7068112 0.00 0.00 >> >> > client_tracing_function >> >> > 0.49 64.96 0.36 1908587 0.00 0.00 multi_runsingle >> >> > >> >> > >> >> > >> >> > test 7: >> >> > enable c-ares; >> >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, -1); >> >> > CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.23.0 >> >> > >> >> > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve >> >> > *.com >> >> > to >> >> > one IP address >> >> > number of domain names in URLs: 780,000 >> >> > number of client: 1000 >> >> > CPU usage: 86% >> >> > cpu% irq% sirq% sys% iowt% mem_used buf&cached >> >> > 86.6 0.0 0.6 3.3 0.0 3624.8Mb 263.0Mb >> >> > SSL TPS: 300 >> >> > SSL throughput: 200Mbps >> >> > >> >> > # gprof /usr/bin/curl-loader gmon.out -p | head -20 >> >> > Flat profile: >> >> > Each sample counts as 0.01 seconds. >> >> > % cumulative self self total >> >> > time seconds seconds calls s/call s/call name >> >> > 39.83 369.71 369.71 207173 0.00 0.00 >> >> > Curl_hash_clean_with_criterium >> >> > 24.27 594.98 225.27 3198337733 0.00 0.00 >> >> > hostcache_timestamp_remove >> >> > 13.75 722.59 127.61 4312864 0.00 0.00 Curl_hash_pick >> >> > 9.19 807.87 85.28 546507 0.00 0.00 Curl_hash_add >> >> > 6.14 864.88 57.01 1022419684 0.00 0.00 >> >> > Curl_str_key_compare >> >> > 2.06 884.00 19.12 203749 0.00 0.00 create_conn >> >> > 0.54 888.99 4.99 213306 0.00 0.00 ConnectionStore >> >> > 0.49 893.50 4.51 7331251 0.00 0.00 dprintf_formatf >> >> > 0.29 896.16 2.66 1287182329 0.00 0.00 >> >> > ares__is_list_empty >> >> > 0.23 898.28 2.12 >> >> > locking_function >> >> > 0.20 900.10 1.82 212210 0.00 0.00 >> >> > Curl_num_addresses >> >> > 0.17 901.66 1.56 198385 0.00 0.00 >> >> > curl_multi_remove_handle >> >> > 0.12 902.81 1.15 411114 0.00 0.00 ares_cancel >> >> > 0.12 903.95 1.14 3683448 0.00 0.00 multi_runsingle >> >> > 0.11 904.97 1.03 13664120 0.00 0.00 >> >> > client_tracing_function >> >> > >> >> > >> >> > test 8: >> >> > enable c-ares; >> >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, 60); >> >> > CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.24.0 >> >> > >> >> > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve >> >> > *.com >> >> > to >> >> > one IP address >> >> > number of domain names in URLs: 780,000 >> >> > number of client: 1000 >> >> > CPU usage: 75% >> >> > cpu% irq% sirq% sys% iowt% mem_used buf&cached >> >> > 75.2 0.0 0.2 2.1 0.0 3537.2Mb 261.4Mb >> >> > SSL TPS: 380 >> >> > SSL throughput: 220Mbps >> >> > >> >> > # gprof /usr/bin/curl-loader gmon.out -p | head -20 >> >> > Flat profile: >> >> > Each sample counts as 0.01 seconds. >> >> > % cumulative self self total >> >> > time seconds seconds calls s/call s/call name >> >> > 27.58 12.31 12.31 71140 0.00 0.00 >> >> > Curl_hash_clean_with_criterium >> >> > 19.09 20.83 8.52 176729692 0.00 0.00 >> >> > hostcache_timestamp_remove >> >> > 10.44 25.49 4.66 1589309 0.00 0.00 Curl_hash_pick >> >> > 6.85 28.55 3.06 73204 0.00 0.00 create_conn >> >> > 5.56 31.03 2.48 67620530 0.00 0.00 >> >> > Curl_str_key_compare >> >> > 4.41 33.00 1.97 200684 0.00 0.00 Curl_hash_add >> >> > 3.56 34.59 1.59 2717956 0.00 0.00 dprintf_formatf >> >> > 1.72 35.36 0.77 75247 0.00 0.00 ConnectionStore >> >> > 1.70 36.12 0.76 444342256 0.00 0.00 >> >> > ares__is_list_empty >> >> > 1.23 36.67 0.55 >> >> > locking_function >> >> > 1.16 37.19 0.52 143342 0.00 0.00 ares_cancel >> >> > 0.99 37.63 0.44 72211 0.00 0.00 >> >> > curl_multi_remove_handle >> >> > 0.74 37.96 0.33 1333371 0.00 0.00 multi_runsingle >> >> > 0.68 38.27 0.31 101147798 0.00 0.00 addbyter >> >> > 0.58 38.53 0.26 288737 0.00 0.00 >> >> > curl_multi_socket_action >> >> > >> >> > >> >> > >> >> > test 9: >> >> > enable c-ares; >> >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, -1); >> >> > CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.24.0 >> >> > >> >> > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve >> >> > *.com >> >> > to >> >> > one IP address >> >> > number of domain names in URLs: 780,000 >> >> > number of client: 1000 >> >> > CPU usage: 75% >> >> > cpu% irq% sirq% sys% iowt% mem_used buf&cached >> >> > 75.1 0.0 0.6 4.1 0.0 3545.2Mb 260.1Mb >> >> > SSL TPS: 380 >> >> > SSL throughput: 200Mbps >> >> > >> >> > # gprof /usr/bin/curl-loader gmon.out -p | head -20 >> >> > Flat profile: >> >> > Each sample counts as 0.01 seconds. >> >> > % cumulative self self total >> >> > time seconds seconds calls s/call s/call name >> >> > 34.00 31.13 31.13 72495 0.00 0.00 >> >> > Curl_hash_clean_with_criterium >> >> > 23.64 52.78 21.65 386070412 0.00 0.00 >> >> > hostcache_timestamp_remove >> >> > 13.28 64.94 12.16 1619440 0.00 0.00 Curl_hash_pick >> >> > 5.92 70.36 5.42 137653129 0.00 0.00 >> >> > Curl_str_key_compare >> >> > 5.50 75.40 5.04 203018 0.00 0.00 Curl_hash_add >> >> > 4.55 79.57 4.17 74618 0.00 0.00 create_conn >> >> > 1.70 81.13 1.56 2757148 0.00 0.00 dprintf_formatf >> >> > 0.98 82.03 0.90 76945 0.00 0.00 ConnectionStore >> >> > 0.95 82.90 0.87 455775100 0.00 0.00 >> >> > ares__is_list_empty >> >> > 0.57 83.42 0.53 >> >> > locking_function >> >> > 0.39 83.78 0.36 146742 0.00 0.00 ares_cancel >> >> > 0.35 84.10 0.32 102968806 0.00 0.00 addbyter >> >> > 0.35 84.42 0.32 456587 0.00 0.00 Curl_readwrite >> >> > 0.35 84.74 0.32 73133 0.00 0.00 >> >> > curl_multi_remove_handle >> >> > 0.34 85.05 0.31 1360545 0.00 0.00 multi_runsingle >> >> > >> >> > >> >> > 2012/3/6 Robert Iakobashvili <cor...@gm...> >> >> >> >> >> >> If it will not be helpful this means that recent >> >> >> changes in libcurl library have broken DNS interface. >> >> >> >> >> >> You can try a previos version and the latest libcurl, like: >> >> >> >> >> >> http://curl.haxx.se/download/curl-7.23.0.tar.gz >> >> >> and >> >> >> http://curl.haxx.se/downloadcurl-7.24.0.tar.gz >> >> >> >> >> >> by: >> >> >> >> >> >> 1. downloading a libcurl version and placing it to >> >> >> curl-loader-0.56/packages ; >> >> >> 2. correcting the Makefile in curl-loader-0.56 >> >> >> CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.24 or CURL_VER:=7.23 >> >> >> 3. make cleanall; make >> >> >> >> >> >> Please, send us your observations. >> >> >> Thanks! >> >> >> Robert >> >> >> >> >> >> On Mon, Mar 5, 2012 at 6:57 PM, Robert Iakobashvili >> >> >> <cor...@gm...> >> >> >> wrote: >> >> >> > If so, please try to change >> >> >> > >> >> >> > loader.c, line 578 which looks like: >> >> >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, -1); >> >> >> > >> >> >> > 1. instead of -1 try to set some value like 60 >> >> >> > 2. make clean; make >> >> >> > >> >> >> > Thanks, >> >> >> > Robert >> >> >> > >> >> >> > On Mon, Mar 5, 2012 at 6:19 PM, Fred Huang <di...@gm...> >> >> >> > wrote: >> >> >> >> >> >> >> >> Looks like the bottleneck comes from libcurl other than c-ares... >> >> >> >> test 3 (no ares patch applied): >> >> >> >> dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve >> >> >> >> *.com >> >> >> >> to one IP address >> >> >> >> number of domain names in URLs: 780,000 >> >> >> >> number of client: 1000 >> >> >> >> CPU usage: 85% >> >> >> >> cpu% irq% sirq% sys% iowt% mem_used buf&cached >> >> >> >> 85.6 0.0 0.8 5.9 0.0 3334.3Mb 112.0Mb >> >> >> >> SSL TPS: 320 >> >> >> >> SSL throughput: 200Mbps >> >> >> >> >> >> >> >> # gprof /usr/bin/curl-loader gmon.out -p | head -50 >> >> >> >> Flat profile: >> >> >> >> Each sample counts as 0.01 seconds. >> >> >> >> % cumulative self self total >> >> >> >> time seconds seconds calls s/call s/call name >> >> >> >> 40.80 120.16 120.16 116079 0.00 0.00 >> >> >> >> Curl_hash_clean_with_criterium >> >> >> >> 24.46 192.18 72.02 983166323 0.00 0.00 >> >> >> >> hostcache_timestamp_remove >> >> >> >> 13.30 231.35 39.17 2059341 0.00 0.00 >> >> >> >> Curl_hash_pick >> >> >> >> 6.45 250.35 19.00 240087 0.00 0.00 >> >> >> >> Curl_hash_add >> >> >> >> 4.81 264.51 14.16 334257102 0.00 0.00 >> >> >> >> Curl_str_key_compare >> >> >> >> 3.40 274.52 10.01 119514 0.00 0.00 create_conn >> >> >> >> 0.93 277.25 2.73 4478910 0.00 0.00 >> >> >> >> dprintf_formatf >> >> >> >> 0.90 279.91 2.66 122209 0.00 0.00 >> >> >> >> ConnectionStore >> >> >> >> 0.40 281.09 1.18 >> >> >> >> locking_function >> >> >> >> 0.27 281.88 0.79 114759 0.00 0.00 >> >> >> >> curl_multi_remove_handle >> >> >> >> 0.21 282.51 0.63 7993664 0.00 0.00 >> >> >> >> client_tracing_function >> >> >> >> 0.21 283.12 0.61 122685 0.00 0.00 >> >> >> >> Curl_num_addresses >> >> >> >> 0.19 283.68 0.56 397028 0.00 0.00 >> >> >> >> curl_multi_socket_action >> >> >> >> 0.19 284.23 0.55 2036116 0.00 0.00 >> >> >> >> multi_runsingle >> >> >> >> 0.17 284.72 0.49 703687 0.00 0.00 >> >> >> >> Curl_readwrite >> >> >> >> 0.16 285.19 0.47 id_function >> >> >> >> 0.15 285.62 0.43 29692029 0.00 0.00 >> >> >> >> Curl_socket_check >> >> >> >> 0.14 286.02 0.40 91973856 0.00 0.00 >> >> >> >> Curl_raw_toupper >> >> >> >> 0.13 286.40 0.38 164664144 0.00 0.00 addbyter >> >> >> >> 0.13 286.77 0.37 2180797 0.00 0.00 Curl_infof >> >> >> >> 0.11 287.10 0.33 122675 0.00 0.00 >> >> >> >> Curl_if_is_interface_name >> >> >> >> 0.10 287.39 0.29 471318 0.00 0.00 >> >> >> >> ossl_connect_common >> >> >> >> 0.10 287.67 0.28 2113528 0.00 0.00 >> >> >> >> Curl_raw_equal >> >> >> >> 0.09 287.93 0.26 8676 0.00 0.02 >> >> >> >> curl_multi_socket_all >> >> >> >> 0.08 288.17 0.24 1663262 0.00 0.00 Curl_splay >> >> >> >> 0.08 288.41 0.24 8505 0.00 0.01 >> >> >> >> curl_multi_perform >> >> >> >> 0.07 288.63 0.22 3114576 0.00 0.00 >> >> >> >> Curl_pgrsUpdate >> >> >> >> 0.06 288.80 0.17 119637 0.00 0.00 >> >> >> >> singleipconnect >> >> >> >> 0.06 288.97 0.17 7994818 0.00 0.00 Curl_debug >> >> >> >> 0.05 289.13 0.16 235820 0.00 0.00 event_del >> >> >> >> 0.05 289.29 0.16 119393071 0.00 0.00 >> >> >> >> curl_strequal >> >> >> >> 0.05 289.45 0.16 18535473 0.00 0.00 >> >> >> >> alloc_addbyter >> >> >> >> 0.05 289.61 0.16 356074 0.00 0.00 >> >> >> >> Curl_hash_str >> >> >> >> 0.05 289.76 0.15 1797233 0.00 0.00 >> >> >> >> fd_key_compare >> >> >> >> 0.05 289.91 0.15 234966 0.00 0.00 >> >> >> >> Curl_ssl_getsessionid >> >> >> >> 0.04 290.04 0.13 585853 0.00 0.00 >> >> >> >> Curl_llist_destroy >> >> >> >> 0.04 290.16 0.12 240826 0.00 0.00 Curl_resolv >> >> >> >> 0.03 290.26 0.10 7514328 0.00 0.00 >> >> >> >> Curl_raw_nequal >> >> >> >> 0.03 290.36 0.10 2214436 0.00 0.00 Curl_read >> >> >> >> 0.03 290.46 0.10 116872 0.00 0.00 >> >> >> >> Curl_http_readwrite_headers >> >> >> >> 0.03 290.56 0.10 >> >> >> >> ssl_tls_trace >> >> >> >> 0.03 290.66 0.10 8013097 0.00 0.00 >> >> >> >> scan_response >> >> >> >> 0.03 290.75 0.09 29488555 0.00 0.00 >> >> >> >> curlx_tvdiff >> >> >> >> 0.03 290.84 0.09 2142468 0.00 0.00 Curl_setopt >> >> >> >> 0.03 290.93 0.09 476299 0.00 0.00 Curl_expire >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> 2012/3/5 Robert Iakobashvili <cor...@gm...> >> >> >> >>> >> >> >> >>> Dear Fred Huang, >> >> >> >>> >> >> >> >>> Thanks for your posting. >> >> >> >>> >> >> >> >>> Could you please check if this is due to some badness at cares? >> >> >> >>> Thanks. >> >> >> >>> >> >> >> >>> Let's exclude usage of cares based DNS resolving by applying the >> >> >> >>> patch >> >> >> >>> attached >> >> >> >>> and use instead global DNS cache. >> >> >> >>> >> >> >> >>> cd ./curl-loader-0.56 >> >> >> >>> patch -p1 < path-to/disable-cares-use-global-dns.patch >> >> >> >>> make cleanall; make >> >> >> >>> >> >> >> >>> See if it helps. >> >> >> >>> >> >> >> >>> Global cache, unfortunately, is not thread safe. >> >> >> >>> >> >> >> >>> If it helps, we'll try to restore the previous version of cares. >> >> >> >>> >> >> >> >>> Thanks, >> >> >> >>> Robert >> >> >> >>> >> >> >> >>> On Mon, Mar 5, 2012 at 4:28 PM, Fred Huang <di...@gm...> >> >> >> >>> wrote: >> >> >> >>>> >> >> >> >>>> test 1: >> >> >> >>>> >> >> >> >>>> dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, >> >> >> >>>> resolve >> >> >> >>>> *.com to one IP address >> >> >> >>>> number of domain names in all https URLs: 780,000 >> >> >> >>>> number of client: 1000 >> >> >> >>>> CPU usage: 78% >> >> >> >>>> cpu% irq% sirq% sys% iowt% mem_used buf&cached >> >> >> >>>> 78.8 0.0 0.6 3.8 0.0 3640.9Mb 282.1Mb >> >> >> >>>> SSL TPS: 300 >> >> >> >>>> SSL throughput: 200Mbps >> >> >> >>>> >> >> >> >>>> # gprof /usr/bin/curl-loader gmon.out -p | head -50 >> >> >> >>>> Flat profile: >> >> >> >>>> Each sample counts as 0.01 seconds. >> >> >> >>>> % cumulative self self total >> >> >> >>>> time seconds seconds calls s/call s/call name >> >> >> >>>> 38.55 127.10 127.10 130975 0.00 0.00 >> >> >> >>>> Curl_hash_clean_with_criterium >> >> >> >>>> 24.63 208.32 81.22 1247528407 0.00 0.00 >> >> >> >>>> hostcache_timestamp_remove >> >> >> >>>> 13.08 251.44 43.12 2791988 0.00 0.00 >> >> >> >>>> Curl_hash_pick >> >> >> >>>> 7.30 275.51 24.07 352490 0.00 0.00 >> >> >> >>>> Curl_hash_add >> >> >> >>>> 6.01 295.31 19.80 410760770 0.00 0.00 >> >> >> >>>> Curl_str_key_compare >> >> >> >>>> 2.95 305.04 9.73 131015 0.00 0.00 >> >> >> >>>> create_conn >> >> >> >>>> 0.88 307.94 2.90 4841390 0.00 0.00 >> >> >> >>>> dprintf_formatf >> >> >> >>>> 0.74 310.38 2.44 136036 0.00 0.00 >> >> >> >>>> ConnectionStore >> >> >> >>>> 0.48 311.96 1.58 814604911 0.00 0.00 >> >> >> >>>> ares__is_list_empty >> >> >> >>>> 0.36 313.16 1.20 >> >> >> >>>> locking_function >> >> >> >>>> 0.27 314.06 0.90 261608 0.00 0.00 >> >> >> >>>> ares_cancel >> >> >> >>>> 0.22 314.79 0.73 127417 0.00 0.00 >> >> >> >>>> curl_multi_remove_handle >> >> >> >>>> 0.19 315.40 0.62 8906301 0.00 0.00 >> >> >> >>>> client_tracing_function >> >> >> >>>> 0.17 315.97 0.57 135305 0.00 0.00 >> >> >> >>>> Curl_num_addresses >> >> >> >>>> 0.17 316.54 0.57 181937907 0.00 0.00 addbyter >> >> >> >>>> 0.17 317.09 0.55 2374363 0.00 0.00 >> >> >> >>>> multi_runsingle >> >> >> >>>> 0.16 317.62 0.53 541954 0.00 0.00 >> >> >> >>>> ares__init_list_node >> >> >> >>>> 0.16 318.14 0.52 494412 0.00 0.00 >> >> >> >>>> curl_multi_socket_action >> >> >> >>>> 0.14 318.60 0.46 33578897 0.00 0.00 >> >> >> >>>> Curl_socket_check >> >> >> >>>> 0.13 319.04 0.44 102951052 0.00 0.00 >> >> >> >>>> Curl_raw_toupper >> >> >> >>>> 0.13 319.48 0.44 787558 0.00 0.00 >> >> >> >>>> Curl_readwrite >> >> >> >>>> 0.13 319.91 0.44 >> >> >> >>>> id_function >> >> >> >>>> 0.12 320.29 0.38 391390 0.00 0.00 >> >> >> >>>> Curl_hash_str >> >> >> >>>> 0.09 320.58 0.29 9216 0.00 0.00 >> >> >> >>>> curl_multi_perform >> >> >> >>>> 0.08 320.86 0.28 2350908 0.00 0.00 >> >> >> >>>> Curl_raw_equal >> >> >> >>>> 0.08 321.14 0.28 2019415 0.00 0.00 >> >> >> >>>> Curl_splay >> >> >> >>>> 0.08 321.40 0.26 520625 0.00 0.00 >> >> >> >>>> ossl_connect_common >> >> >> >>>> 0.08 321.65 0.25 3532992 0.00 0.00 >> >> >> >>>> Curl_pgrsUpdate >> >> >> >>>> 0.07 321.87 0.22 131778537 0.00 0.00 >> >> >> >>>> curl_strequal >> >> >> >>>> 0.05 322.05 0.18 8954118 0.00 0.00 >> >> >> >>>> scan_response >> >> >> >>>> 0.05 322.23 0.18 533644 0.00 0.00 >> >> >> >>>> ares_expand_name >> >> >> >>>> 0.05 322.40 0.17 2477345 0.00 0.00 >> >> >> >>>> fd_key_compare >> >> >> >>>> 0.05 322.57 0.17 2429135 0.00 0.00 >> >> >> >>>> Curl_infof >> >> >> >>>> 0.05 322.74 0.17 130966 0.00 0.00 >> >> >> >>>> singleipconnect >> >> >> >>>> 0.05 322.90 0.16 9292 0.00 0.00 >> >> >> >>>> curl_multi_socket_all >> >> >> >>>> 0.05 323.06 0.16 270137 0.00 0.00 >> >> >> >>>> ares__get_hostent >> >> >> >>>> 0.05 323.21 0.15 8943657 0.00 0.00 >> >> >> >>>> Curl_debug >> >> >> >>>> 0.04 323.35 0.14 262493 0.00 0.00 >> >> >> >>>> Curl_ssl_getsessionid >> >> >> >>>> 0.04 323.48 0.13 566006 0.00 0.00 >> >> >> >>>> socket_callback >> >> >> >>>> 0.04 323.61 0.13 33983864 0.00 0.00 >> >> >> >>>> curlx_tvdiff >> >> >> >>>> 0.04 323.74 0.13 132853 0.00 0.00 >> >> >> >>>> Curl_http_readwrite_headers >> >> >> >>>> 0.04 323.86 0.12 344032 0.00 0.00 epoll_del >> >> >> >>>> 0.04 323.98 0.12 223477 0.00 0.00 Curl_poll >> >> >> >>>> 0.04 324.10 0.12 8526280 0.00 0.00 >> >> >> >>>> Curl_raw_nequal >> >> >> >>>> 0.03 324.21 0.11 348839 0.00 0.00 event_del >> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >> >>>> test 2: >> >> >> >>>> >> >> >> >>>> dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, >> >> >> >>>> resolve >> >> >> >>>> *.com to one IP address >> >> >> >>>> number of domain names in all https URLs: 1 >> >> >> >>>> number of client: 1000 >> >> >> >>>> CPU usage: 75% >> >> >> >>>> SSL TPS: 1300 >> >> >> >>>> SSL throughput: 700Mbps >> >> >> >>>> >> >> >> >>>> # gprof /usr/bin/curl-loader gmon.out -p | head -50 >> >> >> >>>> Flat profile: >> >> >> >>>> Each sample counts as 0.01 seconds. >> >> >> >>>> % cumulative self self total >> >> >> >>>> time seconds seconds calls s/call s/call name >> >> >> >>>> 8.98 2.45 2.45 5127312 0.00 0.00 >> >> >> >>>> dprintf_formatf >> >> >> >>>> 8.90 4.88 2.43 1292796503 0.00 0.00 >> >> >> >>>> ares__is_list_empty >> >> >> >>>> 6.12 6.55 1.67 322461 0.00 0.00 >> >> >> >>>> create_conn >> >> >> >>>> 4.58 7.80 1.25 419872 0.00 0.00 >> >> >> >>>> ares_cancel >> >> >> >>>> 4.10 8.92 1.12 2100162 0.00 0.00 >> >> >> >>>> Curl_readwrite >> >> >> >>>> 3.92 9.99 1.07 4527388 0.00 0.00 >> >> >> >>>> Curl_hash_pick >> >> >> >>>> 3.43 10.93 0.94 >> >> >> >>>> locking_function >> >> >> >>>> 3.04 11.76 0.83 909759 0.00 0.00 >> >> >> >>>> curl_multi_socket_action >> >> >> >>>> 3.00 12.58 0.82 97252 0.00 0.00 >> >> >> >>>> Curl_hash_clean_with_criterium >> >> >> >>>> 2.71 13.32 0.74 179704016 0.00 0.00 >> >> >> >>>> Curl_raw_toupper >> >> >> >>>> 2.68 14.05 0.73 4646524 0.00 0.00 >> >> >> >>>> multi_runsingle >> >> >> >>>> 2.44 14.71 0.67 13862992 0.00 0.00 >> >> >> >>>> client_tracing_function >> >> >> >>>> 2.42 15.37 0.66 319735 0.00 0.00 >> >> >> >>>> curl_multi_remove_handle >> >> >> >>>> 2.42 16.03 0.66 22 0.03 0.03 >> >> >> >>>> ares__init_list_node >> >> >> >>>> 1.94 16.56 0.53 9930244 0.00 0.00 >> >> >> >>>> hostcache_timestamp_remove >> >> >> >>>> 1.94 17.09 0.53 169320088 0.00 0.00 addbyter >> >> >> >>>> 1.80 17.58 0.49 >> >> >> >>>> id_function >> >> >> >>>> 1.36 17.95 0.37 7057146 0.00 0.00 >> >> >> >>>> Curl_pgrsUpdate >> >> >> >>>> 1.36 18.32 0.37 16835 0.00 0.00 >> >> >> >>>> curl_multi_socket_all >> >> >> >>>> 1.25 18.66 0.34 16719 0.00 0.00 >> >> >> >>>> curl_multi_perform >> >> >> >>>> 1.21 18.99 0.33 329403 0.00 0.00 >> >> >> >>>> Curl_http_readwrite_headers >> >> >> >>>> 1.17 19.31 0.32 4036624 0.00 0.00 >> >> >> >>>> Curl_splay >> >> >> >>>> 1.14 19.62 0.31 20838902 0.00 0.00 >> >> >> >>>> Curl_raw_nequal >> >> >> >>>> 0.82 19.85 0.23 2764123 0.00 0.00 >> >> >> >>>> Curl_infof >> >> >> >>>> 0.81 20.07 0.22 5933857 0.00 0.00 ossl_recv >> >> >> >>>> 0.77 20.28 0.21 323112 0.00 0.00 >> >> >> >>>> Curl_splayremovebyaddr >> >> >> >>>> 0.75 20.48 0.21 5943991 0.00 0.00 Curl_read >> >> >> >>>> 0.70 20.67 0.19 425174 0.00 0.00 event_del >> >> >> >>>> 0.70 20.86 0.19 98652 0.00 0.00 >> >> >> >>>> Curl_if_is_interface_name >> >> >> >>>> 0.70 21.05 0.19 8677733 0.00 0.00 >> >> >> >>>> Curl_socket_check >> >> >> >>>> 0.59 21.21 0.16 4129554 0.00 0.00 >> >> >> >>>> fd_key_compare >> >> >> >>>> 0.57 21.37 0.16 14017019 0.00 0.00 >> >> >> >>>> Curl_debug >> >> >> >>>> 0.55 21.52 0.15 9376861 0.00 0.00 >> >> >> >>>> stat_data_in_add >> >> >> >>>> 0.55 21.67 0.15 6840468 0.00 0.00 >> >> >> >>>> Curl_timeleft >> >> >> >>>> 0.55 21.82 0.15 2054602 0.00 0.00 >> >> >> >>>> Curl_raw_equal >> >> >> >>>> 0.55 21.97 0.15 1291611 0.00 0.00 >> >> >> >>>> Curl_expire >> >> >> >>>> 0.51 22.11 0.14 16570063 0.00 0.00 >> >> >> >>>> curlx_tvnow >> >> >> >>>> 0.51 22.25 0.14 13863284 0.00 0.00 >> >> >> >>>> scan_response >> >> >> >>>> 0.51 22.39 0.14 5498864 0.00 0.00 >> >> >> >>>> Curl_setopt >> >> >> >>>> 0.48 22.52 0.13 97490291 0.00 0.00 >> >> >> >>>> curl_strequal >> >> >> >>>> 0.44 22.64 0.12 324253 0.00 0.00 Curl_http >> >> >> >>>> 0.44 22.76 0.12 394386 0.00 0.00 >> >> >> >>>> ossl_connect_common >> >> >> >>>> 0.40 22.87 0.11 3581335 0.00 0.00 >> >> >> >>>> Curl_getinfo >> >> >> >>>> 0.38 22.97 0.11 26983092 0.00 0.00 >> >> >> >>>> alloc_addbyter >> >> >> >>>> 0.37 23.07 0.10 8729044 0.00 0.00 >> >> >> >>>> Curl_client_write >> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >> >>>> ------------------------------------------------------------------------------ >> >> >> >>>> Try before you buy = See our experts in action! >> >> >> >>>> The most comprehensive online learning library for Microsoft >> >> >> >>>> developers >> >> >> >>>> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, >> >> >> >>>> CSS3, >> >> >> >>>> MVC3, >> >> >> >>>> Metro Style Apps, more. Free future releases when you subscribe >> >> >> >>>> now! >> >> >> >>>> http://p.sf.net/sfu/learndevnow-dev2 >> >> >> >>>> _______________________________________________ >> >> >> >>>> curl-loader-devel mailing list >> >> >> >>>> cur...@li... >> >> >> >>>> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel >> >> >> >>>> >> >> >> >>> >> >> >> >>> >> >> >> >>> >> >> >> >>> -- >> >> >> >>> Regards, >> >> >> >>> Robert Iakobashvili, Ph.D. >> >> >> >>> >> >> >> >>> Home: http://www.ghotit.com >> >> >> >>> Blog: http://www.ghotit.com/category/ghotit-blogs/ >> >> >> >>> Twitter: http://twitter.com/ghotit >> >> >> >>> Facebook: http://facebook.com/ghotit >> >> >> >>> >> >> >> >>> >> >> >> >>> ...................................................................... >> >> >> >>> Ghotit Dyslexia >> >> >> >>> Assistive technology that understands you >> >> >> >>> >> >> >> >>> >> >> >> >>> ...................................................................... >> >> >> >>> >> >> >> >>> >> >> >> >>> >> >> >> >>> >> >> >> >>> >> >> >> >>> ------------------------------------------------------------------------------ >> >> >> >>> Try before you buy = See our experts in action! >> >> >> >>> The most comprehensive online learning library for Microsoft >> >> >> >>> developers >> >> >> >>> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, >> >> >> >>> CSS3, >> >> >> >>> MVC3, >> >> >> >>> Metro Style Apps, more. Free future releases when you subscribe >> >> >> >>> now! >> >> >> >>> http://p.sf.net/sfu/learndevnow-dev2 >> >> >> >>> _______________________________________________ >> >> >> >>> curl-loader-devel mailing list >> >> >> >>> cur...@li... >> >> >> >>> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel >> >> >> >>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> ------------------------------------------------------------------------------ >> >> >> >> Try before you buy = See our experts in action! >> >> >> >> The most comprehensive online learning library for Microsoft >> >> >> >> developers >> >> >> >> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, >> >> >> >> CSS3, >> >> >> >> MVC3, >> >> >> >> Metro Style Apps, more. Free future releases when you subscribe >> >> >> >> now! >> >> >> >> http://p.sf.net/sfu/learndevnow-dev2 >> >> >> >> _______________________________________________ >> >> >> >> curl-loader-devel mailing list >> >> >> >> cur...@li... >> >> >> >> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel >> >> >> >> >> >> >> > >> >> >> > >> >> >> > >> >> >> > -- >> >> >> > Regards, >> >> >> > Robert Iakobashvili, Ph.D. >> >> >> > >> >> >> > Home: http://www.ghotit.com >> >> >> > Blog: http://www.ghotit.com/category/ghotit-blogs/ >> >> >> > Twitter: http://twitter.com/ghotit >> >> >> > Facebook: http://facebook.com/ghotit >> >> >> > >> >> >> > >> >> >> > ...................................................................... >> >> >> > Ghotit Dyslexia >> >> >> > Assistive technology that understands you >> >> >> > >> >> >> > >> >> >> > ...................................................................... >> >> >> >> >> >> >> >> >> >> >> >> -- >> >> >> Regards, >> >> >> Robert Iakobashvili, Ph.D. >> >> >> >> >> >> Home: http://www.ghotit.com >> >> >> Blog: http://www.ghotit.com/category/ghotit-blogs/ >> >> >> Twitter: http://twitter.com/ghotit >> >> >> Facebook: http://facebook.com/ghotit >> >> >> >> >> >> ...................................................................... >> >> >> Ghotit Dyslexia >> >> >> Assistive technology that understands you >> >> >> >> >> >> ...................................................................... >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> ------------------------------------------------------------------------------ >> >> >> Try before you buy = See our experts in action! >> >> >> The most comprehensive online learning library for Microsoft >> >> >> developers >> >> >> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, >> >> >> MVC3, >> >> >> Metro Style Apps, more. Free future releases when you subscribe now! >> >> >> http://p.sf.net/sfu/learndevnow-dev2 >> >> >> _______________________________________________ >> >> >> curl-loader-devel mailing list >> >> >> cur...@li... >> >> >> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > ------------------------------------------------------------------------------ >> >> > Keep Your Developer Skills Current with LearnDevNow! >> >> > The most comprehensive online learning library for Microsoft >> >> > developers >> >> > is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, >> >> > MVC3, >> >> > Metro Style Apps, more. Free future releases when you subscribe now! >> >> > http://p.sf.net/sfu/learndevnow-d2d >> >> > _______________________________________________ >> >> > curl-loader-devel mailing list >> >> > cur...@li... >> >> > https://lists.sourceforge.net/lists/listinfo/curl-loader-devel >> >> > >> >> >> >> >> >> >> >> -- >> >> Regards, >> >> Robert Iakobashvili, Ph.D. >> >> >> >> Home: http://www.ghotit.com >> >> Blog: http://www.ghotit.com/category/ghotit-blogs/ >> >> Twitter: http://twitter.com/ghotit >> >> Facebook: http://facebook.com/ghotit >> >> ...................................................................... >> >> Ghotit Dyslexia >> >> Assistive technology that understands you >> >> ...................................................................... >> >> >> >> >> >> >> >> ------------------------------------------------------------------------------ >> >> Keep Your Developer Skills Current with LearnDevNow! >> >> The most comprehensive online learning library for Microsoft developers >> >> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, >> >> MVC3, >> >> Metro Style Apps, more. Free future releases when you subscribe now! >> >> http://p.sf.net/sfu/learndevnow-d2d >> >> _______________________________________________ >> >> curl-loader-devel mailing list >> >> cur...@li... >> >> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel >> > >> > >> > >> > >> > ------------------------------------------------------------------------------ >> > Keep Your Developer Skills Current with LearnDevNow! >> > The most comprehensive online learning library for Microsoft developers >> > is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, >> > Metro Style Apps, more. Free future releases when you subscribe now! >> > http://p.sf.net/sfu/learndevnow-d2d >> > _______________________________________________ >> > curl-loader-devel mailing list >> > cur...@li... >> > https://lists.sourceforge.net/lists/listinfo/curl-loader-devel >> > >> >> >> >> -- >> Regards, >> Robert Iakobashvili, Ph.D. >> >> Home: http://www.ghotit.com >> Blog: http://www.ghotit.com/category/ghotit-blogs/ >> Twitter: http://twitter.com/ghotit >> Facebook: http://facebook.com/ghotit >> ...................................................................... >> Ghotit Dyslexia >> Assistive technology that understands you >> ...................................................................... >> >> >> ------------------------------------------------------------------------------ >> Keep Your Developer Skills Current with LearnDevNow! >> The most comprehensive online learning library for Microsoft developers >> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, >> Metro Style Apps, more. Free future releases when you subscribe now! >> http://p.sf.net/sfu/learndevnow-d2d >> _______________________________________________ >> curl-loader-devel mailing list >> cur...@li... >> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel > > > > ------------------------------------------------------------------------------ > Virtualization & Cloud Management Using Capacity Planning > Cloud computing makes use of virtualization - but cloud computing > also focuses on allowing computing to be delivered as a service. > http://www.accelacomm.com/jaw/sfnl/114/51521223/ > _______________________________________________ > curl-loader-devel mailing list > cur...@li... > https://lists.sourceforge.net/lists/listinfo/curl-loader-devel > -- Regards, Robert Iakobashvili, Ph.D. Home: http://www.ghotit.com Blog: http://www.ghotit.com/category/ghotit-blogs/ Twitter: http://twitter.com/ghotit Facebook: http://facebook.com/ghotit ...................................................................... Ghotit Dyslexia Assistive technology that understands you ...................................................................... |
|
From: Robert I. <cor...@gm...> - 2012-03-07 06:34:28
|
Sorry, Fred. I'm poised to think that thread-locking with openssl is not the issue here but rather libcurl interaction with openssl is to blame. The fix could be done only in libcurl or even in openssl. If you could manage to compile openssl with -pg -O2 and to pass these openssl binaries to linking, perhaps, you could see more in your profiling. Sincerely, Robert On Wed, Mar 7, 2012 at 2:43 AM, Fred Huang <di...@gm...> wrote: > yes, I use -t 8 option because the test box has 8 logic CPUs (Dell 990, > Intel Core i7 2600, 16GB memory). > if I do not use -t option, the CPU used by curl-loader will be 100% on one > of 8 logic CPU (12% of total CPU), and the CAPS drops to 100: > # gprof /usr/bin/curl-loader gmon.out -p | head -20 > Flat profile: > Each sample counts as 0.01 seconds. > % cumulative self self total > time seconds seconds calls s/call s/call name > 34.12 2.74 2.74 17385 0.00 0.00 create_conn > 11.21 3.64 0.90 30998583 0.00 0.00 > hostcache_timestamp_remove > 9.84 4.43 0.79 13433 0.00 0.00 > Curl_hash_clean_with_criterium > 7.97 5.07 0.64 381818 0.00 0.00 Curl_hash_pick > 5.85 5.54 0.47 17385 0.00 0.00 ConnectionStore > 5.11 5.95 0.41 16385 0.00 0.00 > curl_multi_remove_handle > 4.11 6.28 0.33 16509889 0.00 0.00 Curl_str_key_compare > 3.36 6.55 0.27 23959914 0.00 0.00 Curl_socket_check > 3.24 6.81 0.26 56703 0.00 0.00 Curl_hash_add > 2.62 7.02 0.21 635750 0.00 0.00 dprintf_formatf > 1.25 7.12 0.10 92421190 0.00 0.00 ares__is_list_empty > 0.75 7.18 0.06 97957 0.00 0.00 Curl_readwrite > 0.62 7.23 0.05 23422260 0.00 0.00 addbyter > 0.62 7.28 0.05 1136775 0.00 0.00 > client_tracing_function > 0.50 7.32 0.04 3091375 0.00 0.00 ares__init_list_head > > I also tried to use > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, 20); > but get no improvement on CAPS. > > > Regards, > Fred > > 2012/3/7 Robert Iakobashvili <cor...@gm...> >> >> Dear Fred, >> >> If I understand correctly your observations, the issue is not in: >> >> 1. DNS resolving; >> 2. Version of cares or libcurl; >> >> but rather in >> >> Less effective handling of HTTPS versus HTTP in libcurl. >> >> Please, confirm that my understanding is correct. >> >> >> HTTPS URLs are using an extra locking in openssl with -t option >> that could be making the issue. >> If you are using -t option, try to load without it and let's say with >> 300 clients. >> >> If you are not using -t, thus the only way to proceed is to share >> your observations with libcurl people at >> >> libcurl development <cur...@co...> >> >> Thank you. >> >> Best wishes. >> Robert >> >> On Tue, Mar 6, 2012 at 6:17 PM, Fred Huang <di...@gm...> wrote: >> > 1. the major CPU consumer is curl-loader. here is the top: >> > top - 07:36:36 up 23:31, 5 users, load average: 4.32, 2.26, 0.91 >> > Tasks: 237 total, 2 running, 235 sleeping, 0 stopped, 0 zombie >> > Cpu0 : 76.6%us, 3.6%sy, 0.0%ni, 19.4%id, 0.0%wa, 0.0%hi, 0.3%si, >> > 0.0% >> > Cpu1 : 72.4%us, 4.0%sy, 0.0%ni, 22.9%id, 0.0%wa, 0.0%hi, 0.7%si, >> > 0.0% >> > Cpu2 : 65.8%us, 3.3%sy, 0.0%ni, 30.6%id, 0.0%wa, 0.0%hi, 0.3%si, >> > 0.0% >> > Cpu3 : 75.2%us, 4.3%sy, 0.0%ni, 19.5%id, 0.0%wa, 0.0%hi, 1.0%si, >> > 0.0% >> > Cpu4 : 79.1%us, 4.0%sy, 0.0%ni, 16.8%id, 0.0%wa, 0.0%hi, 0.0%si, >> > 0.0% >> > Cpu5 : 75.8%us, 4.3%sy, 0.0%ni, 19.2%id, 0.0%wa, 0.0%hi, 0.7%si, >> > 0.0% >> > Cpu6 : 74.1%us, 5.0%sy, 0.0%ni, 20.3%id, 0.0%wa, 0.0%hi, 0.7%si, >> > 0.0% >> > Cpu7 : 78.7%us, 4.0%sy, 0.0%ni, 16.6%id, 0.0%wa, 0.0%hi, 0.7%si, >> > 0.0% >> > Mem: 16209932k total, 3829500k used, 12380432k free, 99112k buffers >> > Swap: 18448376k total, 0k used, 18448376k free, 320284k cached >> > >> > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND >> > 30795 root 20 0 3200m 2.8g 2296 S 79.2 18.4 21:26.84 curl-loader >> > 1832 nobody 20 0 13944 1736 580 S 0.1 0.0 0:32.65 dnsmasq >> > 9 root 20 0 0 0 0 S 0.0 0.0 0:00.53 ksoftirqd/1 >> > 30825 root 20 0 15220 1360 960 R 0.0 0.0 0:00.42 top >> > 1 root 20 0 19400 1568 1256 S 0.0 0.0 0:00.75 init >> > 2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd >> > 3 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/0 >> > 4 root 20 0 0 0 0 S 0.0 0.0 0:00.06 ksoftirqd/0 >> > 5 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/0 >> > >> > >> > 2. 1000 loading clients >> > >> > 3. yes, tried 0.52 but no lucky -- CAPS 380, throughput 200Mbps, CPU 90% >> > top - 08:17:10 up 1 day, 12 min, 8 users, load average: 4.66, 3.39, >> > 1.88 >> > Tasks: 245 total, 2 running, 243 sleeping, 0 stopped, 0 zombie >> > Cpu0 : 82.2%us, 4.0%sy, 0.0%ni, 13.9%id, 0.0%wa, 0.0%hi, 0.0%si, >> > 0.0% >> > Cpu1 : 86.1%us, 4.3%sy, 0.0%ni, 7.9%id, 0.0%wa, 0.0%hi, 1.7%si, >> > 0.0% >> > Cpu2 : 85.3%us, 4.0%sy, 0.0%ni, 10.0%id, 0.0%wa, 0.0%hi, 0.7%si, >> > 0.0% >> > Cpu3 : 81.8%us, 3.0%sy, 0.0%ni, 14.2%id, 0.0%wa, 0.0%hi, 1.0%si, >> > 0.0% >> > Cpu4 : 83.7%us, 4.0%sy, 0.0%ni, 12.3%id, 0.0%wa, 0.0%hi, 0.0%si, >> > 0.0% >> > Cpu5 : 92.7%us, 3.3%sy, 0.0%ni, 3.3%id, 0.0%wa, 0.0%hi, 0.7%si, >> > 0.0% >> > Cpu6 : 89.7%us, 2.0%sy, 0.0%ni, 8.0%id, 0.0%wa, 0.0%hi, 0.3%si, >> > 0.0% >> > Cpu7 : 88.1%us, 3.6%sy, 0.0%ni, 6.6%id, 0.0%wa, 0.0%hi, 1.7%si, >> > 0.0% >> > Mem: 16209932k total, 1795748k used, 14414184k free, 102824k buffers >> > Swap: 18448376k total, 0k used, 18448376k free, 382988k cached >> > >> > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND >> > 12236 root 20 0 1150m 894m 2572 S 90.7 5.7 51:11.91 curl-loader >> > 1832 nobody 20 0 13944 1736 580 S 0.1 0.0 0:47.04 dnsmasq >> > >> > ]# gprof /usr/bin/curl-loader gmon.out -p | head -20 >> > Flat profile: >> > Each sample counts as 0.01 seconds. >> > % cumulative self self total >> > time seconds seconds calls s/call s/call name >> > 43.92 151.95 151.95 157295 0.00 0.00 >> > Curl_hash_clean_with_criterium >> > 22.90 231.17 79.22 1712190971 0.00 0.00 >> > hostcache_timestamp_remove >> > 13.15 276.68 45.51 3260321 0.00 0.00 Curl_hash_pick >> > 7.51 302.65 25.97 394171 0.00 0.00 Curl_hash_add >> > 5.98 323.34 20.68 567437518 0.00 0.00 >> > Curl_str_key_compare >> > 1.00 326.80 3.46 5380220 0.00 0.00 dprintf_formatf >> > 0.41 328.21 1.41 locking_function >> > 0.32 329.32 1.11 469222138 0.00 0.00 >> > ares__is_list_empty >> > 0.31 330.40 1.08 3284508 0.00 0.00 multi_runsingle >> > 0.28 331.38 0.98 10006806 0.00 0.00 >> > client_tracing_function >> > 0.24 332.21 0.83 151951 0.00 0.00 Curl_num_addresses >> > 0.22 332.97 0.77 188075672 0.00 0.00 addbyter >> > 0.19 333.64 0.67 841840 0.00 0.00 Curl_readwrite >> > 0.19 334.29 0.65 id_function >> > 0.18 334.90 0.61 39843 0.00 0.01 >> > curl_multi_socket_all >> > >> > >> > >> > 4. yes, it is correct for 0.56. If use same URL list by replacing https >> > to >> > http, I can get CAPS 1500 and throughput 870Mbps with CPU% < 60% >> > >> > >> > 2012/3/6 Robert Iakobashvili <cor...@gm...> >> >> >> >> Several questions: >> >> >> >> 1. Could you please run top and see if there are other processes >> >> burning >> >> CPU? >> >> 2. What is the number of loading clients that you use? >> >> 3. Could you try curl-loader-0.52 if it compiles for you? >> >> If not, please try to place curl-loader-7.19.7.tar.gz and >> >> c-ares-16.0.tar.gz, >> >> update their versions in Makefile and rebuild by make cleanall; make >> >> >> >> 4. Is it correct to say that for HTTP URLs (not HTTPS) you are getting >> >> a good throughput >> >> and CAPS with curl-loader-0.56? >> >> >> >> Thanks, >> >> Robert >> >> >> >> On Tue, Mar 6, 2012 at 3:52 AM, Fred Huang <di...@gm...> wrote: >> >> > TIMEOUT -1 -> 60 works a little better but no performance improvement >> >> > on >> >> > CAPS or throughput. >> >> > >> >> > test 5: >> >> > enable c-ares; >> >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, 60); >> >> > >> >> > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve >> >> > *.com >> >> > to >> >> > one IP address >> >> > number of domain names in URLs: 780,000 >> >> > number of client: 1000 >> >> > CPU usage: 74% >> >> > cpu% irq% sirq% sys% iowt% mem_used buf&cached >> >> > 74.0 0.0 0.7 4.2 0.0 3530.6Mb 250.8Mb >> >> > SSL TPS: 360 >> >> > SSL throughput: 220Mbps >> >> > >> >> > # gprof /usr/bin/curl-loader gmon.out -p | head -50 >> >> > Flat profile: >> >> > Each sample counts as 0.01 seconds. >> >> > % cumulative self self total >> >> > time seconds seconds calls s/call s/call name >> >> > 26.63 23.13 23.13 119226 0.00 0.00 >> >> > Curl_hash_clean_with_criterium >> >> > 20.34 40.80 17.67 317792146 0.00 0.00 >> >> > hostcache_timestamp_remove >> >> > 11.27 50.59 9.79 2579903 0.00 0.00 Curl_hash_pick >> >> > 7.84 57.40 6.81 120606 0.00 0.00 create_conn >> >> > 5.65 62.31 4.91 119117401 0.00 0.00 >> >> > Curl_str_key_compare >> >> > 4.98 66.64 4.33 329829 0.00 0.00 Curl_hash_add >> >> > 2.80 69.08 2.44 4476606 0.00 0.00 dprintf_formatf >> >> > 2.57 71.31 2.23 124019 0.00 0.00 ConnectionStore >> >> > 1.50 72.61 1.30 740997609 0.00 0.00 >> >> > ares__is_list_empty >> >> > 1.16 73.61 1.01 >> >> > locking_function >> >> > 0.76 74.27 0.66 238744 0.00 0.00 ares_cancel >> >> > 0.70 74.88 0.61 457564 0.00 0.00 >> >> > curl_multi_socket_action >> >> > 0.67 75.46 0.58 118616 0.00 0.00 >> >> > curl_multi_remove_handle >> >> > 0.58 75.97 0.51 8211502 0.00 0.00 >> >> > client_tracing_function >> >> > 0.53 76.43 0.46 733561 0.00 0.00 Curl_readwrite >> >> > 0.51 76.87 0.44 2183179 0.00 0.00 multi_runsingle >> >> > 0.51 77.31 0.44 166585567 0.00 0.00 addbyter >> >> > 0.44 77.69 0.38 2175805 0.00 0.00 Curl_raw_equal >> >> > 0.43 78.06 0.38 id_function >> >> > 0.40 78.41 0.35 29789346 0.00 0.00 >> >> > Curl_socket_check >> >> > 0.40 78.76 0.35 479671 0.00 0.00 >> >> > ossl_connect_common >> >> > 0.40 79.11 0.35 490639 0.00 0.00 >> >> > ares__init_list_node >> >> > 0.33 79.40 0.29 94197670 0.00 0.00 >> >> > Curl_raw_toupper >> >> > 0.33 79.69 0.29 3268031 0.00 0.00 Curl_pgrsUpdate >> >> > 0.28 79.93 0.24 8797 0.00 0.00 >> >> > curl_multi_perform >> >> > 0.27 80.17 0.24 2233641 0.00 0.00 Curl_infof >> >> > 0.26 80.40 0.23 1839837 0.00 0.00 Curl_splay >> >> > 0.24 80.61 0.21 121713028 0.00 0.00 curl_strequal >> >> > 0.21 80.79 0.18 243333 0.00 0.00 >> >> > Curl_ssl_getsessionid >> >> > 0.18 80.95 0.16 122180 0.00 0.00 >> >> > Curl_num_addresses >> >> > 0.17 81.09 0.15 8271666 0.00 0.00 Curl_debug >> >> > 0.16 81.23 0.14 18902452 0.00 0.00 alloc_addbyter >> >> > 0.16 81.37 0.14 8851 0.00 0.00 >> >> > curl_multi_socket_all >> >> > 0.14 81.50 0.13 2236769 0.00 0.00 Curl_read >> >> > 0.14 81.62 0.12 2271700 0.00 0.00 fd_key_compare >> >> > 0.14 81.74 0.12 323167 0.00 0.00 event_del >> >> > 0.14 81.86 0.12 205436 0.00 0.00 >> >> > read_udp_packets >> >> > 0.14 81.98 0.12 200979 0.00 0.00 processfds >> >> > 0.13 82.09 0.11 123886 0.00 0.00 >> >> > Curl_if_is_interface_name >> >> > 0.13 82.20 0.11 120350 0.00 0.00 Curl_http >> >> > 0.13 82.31 0.11 ssl_tls_trace >> >> > 0.12 82.41 0.10 119229 0.00 0.00 >> >> > Curl_splayremovebyaddr >> >> > 0.12 82.51 0.10 7798717 0.00 0.00 Curl_raw_nequal >> >> > 0.12 82.61 0.10 784485 0.00 0.00 event_add >> >> > 0.10 82.70 0.09 643482 0.00 0.00 Curl_expire >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > test 6: >> >> > enable c-ares; >> >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, 60); >> >> > CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.23.0 >> >> > >> >> > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve >> >> > *.com >> >> > to >> >> > one IP address >> >> > number of domain names in URLs: 780,000 >> >> > number of client: 1000 >> >> > CPU usage: 86% >> >> > cpu% irq% sirq% sys% iowt% mem_used buf&cached >> >> > 85.9 0.0 0.2 2.1 0.0 3537.2Mb 261.4Mb >> >> > SSL TPS: 360 >> >> > SSL throughput: 200Mbps >> >> > >> >> > # gprof /usr/bin/curl-loader gmon.out -p | head -20 >> >> > Flat profile: >> >> > Each sample counts as 0.01 seconds. >> >> > % cumulative self self total >> >> > time seconds seconds calls s/call s/call name >> >> > 27.19 19.96 19.96 103472 0.00 0.00 >> >> > Curl_hash_clean_with_criterium >> >> > 21.73 35.91 15.95 274168981 0.00 0.00 >> >> > hostcache_timestamp_remove >> >> > 10.33 43.49 7.58 2256441 0.00 0.00 Curl_hash_pick >> >> > 7.21 48.78 5.29 104540 0.00 0.00 create_conn >> >> > 6.20 53.33 4.55 103413405 0.00 0.00 >> >> > Curl_str_key_compare >> >> > 4.88 56.91 3.58 287109 0.00 0.00 Curl_hash_add >> >> > 3.02 59.13 2.22 3797855 0.00 0.00 dprintf_formatf >> >> > 2.10 60.67 1.54 107711 0.00 0.00 ConnectionStore >> >> > 1.65 61.88 1.21 642725327 0.00 0.00 >> >> > ares__is_list_empty >> >> > 1.19 62.76 0.88 >> >> > locking_function >> >> > 0.87 63.40 0.64 207817 0.00 0.00 ares_cancel >> >> > 0.56 63.81 0.41 404108 0.00 0.00 >> >> > curl_multi_socket_action >> >> > 0.54 64.21 0.40 425603 0.00 0.00 >> >> > ares__init_list_node >> >> > 0.53 64.60 0.39 7068112 0.00 0.00 >> >> > client_tracing_function >> >> > 0.49 64.96 0.36 1908587 0.00 0.00 multi_runsingle >> >> > >> >> > >> >> > >> >> > test 7: >> >> > enable c-ares; >> >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, -1); >> >> > CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.23.0 >> >> > >> >> > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve >> >> > *.com >> >> > to >> >> > one IP address >> >> > number of domain names in URLs: 780,000 >> >> > number of client: 1000 >> >> > CPU usage: 86% >> >> > cpu% irq% sirq% sys% iowt% mem_used buf&cached >> >> > 86.6 0.0 0.6 3.3 0.0 3624.8Mb 263.0Mb >> >> > SSL TPS: 300 >> >> > SSL throughput: 200Mbps >> >> > >> >> > # gprof /usr/bin/curl-loader gmon.out -p | head -20 >> >> > Flat profile: >> >> > Each sample counts as 0.01 seconds. >> >> > % cumulative self self total >> >> > time seconds seconds calls s/call s/call name >> >> > 39.83 369.71 369.71 207173 0.00 0.00 >> >> > Curl_hash_clean_with_criterium >> >> > 24.27 594.98 225.27 3198337733 0.00 0.00 >> >> > hostcache_timestamp_remove >> >> > 13.75 722.59 127.61 4312864 0.00 0.00 Curl_hash_pick >> >> > 9.19 807.87 85.28 546507 0.00 0.00 Curl_hash_add >> >> > 6.14 864.88 57.01 1022419684 0.00 0.00 >> >> > Curl_str_key_compare >> >> > 2.06 884.00 19.12 203749 0.00 0.00 create_conn >> >> > 0.54 888.99 4.99 213306 0.00 0.00 ConnectionStore >> >> > 0.49 893.50 4.51 7331251 0.00 0.00 dprintf_formatf >> >> > 0.29 896.16 2.66 1287182329 0.00 0.00 >> >> > ares__is_list_empty >> >> > 0.23 898.28 2.12 >> >> > locking_function >> >> > 0.20 900.10 1.82 212210 0.00 0.00 >> >> > Curl_num_addresses >> >> > 0.17 901.66 1.56 198385 0.00 0.00 >> >> > curl_multi_remove_handle >> >> > 0.12 902.81 1.15 411114 0.00 0.00 ares_cancel >> >> > 0.12 903.95 1.14 3683448 0.00 0.00 multi_runsingle >> >> > 0.11 904.97 1.03 13664120 0.00 0.00 >> >> > client_tracing_function >> >> > >> >> > >> >> > test 8: >> >> > enable c-ares; >> >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, 60); >> >> > CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.24.0 >> >> > >> >> > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve >> >> > *.com >> >> > to >> >> > one IP address >> >> > number of domain names in URLs: 780,000 >> >> > number of client: 1000 >> >> > CPU usage: 75% >> >> > cpu% irq% sirq% sys% iowt% mem_used buf&cached >> >> > 75.2 0.0 0.2 2.1 0.0 3537.2Mb 261.4Mb >> >> > SSL TPS: 380 >> >> > SSL throughput: 220Mbps >> >> > >> >> > # gprof /usr/bin/curl-loader gmon.out -p | head -20 >> >> > Flat profile: >> >> > Each sample counts as 0.01 seconds. >> >> > % cumulative self self total >> >> > time seconds seconds calls s/call s/call name >> >> > 27.58 12.31 12.31 71140 0.00 0.00 >> >> > Curl_hash_clean_with_criterium >> >> > 19.09 20.83 8.52 176729692 0.00 0.00 >> >> > hostcache_timestamp_remove >> >> > 10.44 25.49 4.66 1589309 0.00 0.00 Curl_hash_pick >> >> > 6.85 28.55 3.06 73204 0.00 0.00 create_conn >> >> > 5.56 31.03 2.48 67620530 0.00 0.00 >> >> > Curl_str_key_compare >> >> > 4.41 33.00 1.97 200684 0.00 0.00 Curl_hash_add >> >> > 3.56 34.59 1.59 2717956 0.00 0.00 dprintf_formatf >> >> > 1.72 35.36 0.77 75247 0.00 0.00 ConnectionStore >> >> > 1.70 36.12 0.76 444342256 0.00 0.00 >> >> > ares__is_list_empty >> >> > 1.23 36.67 0.55 >> >> > locking_function >> >> > 1.16 37.19 0.52 143342 0.00 0.00 ares_cancel >> >> > 0.99 37.63 0.44 72211 0.00 0.00 >> >> > curl_multi_remove_handle >> >> > 0.74 37.96 0.33 1333371 0.00 0.00 multi_runsingle >> >> > 0.68 38.27 0.31 101147798 0.00 0.00 addbyter >> >> > 0.58 38.53 0.26 288737 0.00 0.00 >> >> > curl_multi_socket_action >> >> > >> >> > >> >> > >> >> > test 9: >> >> > enable c-ares; >> >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, -1); >> >> > CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.24.0 >> >> > >> >> > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve >> >> > *.com >> >> > to >> >> > one IP address >> >> > number of domain names in URLs: 780,000 >> >> > number of client: 1000 >> >> > CPU usage: 75% >> >> > cpu% irq% sirq% sys% iowt% mem_used buf&cached >> >> > 75.1 0.0 0.6 4.1 0.0 3545.2Mb 260.1Mb >> >> > SSL TPS: 380 >> >> > SSL throughput: 200Mbps >> >> > >> >> > # gprof /usr/bin/curl-loader gmon.out -p | head -20 >> >> > Flat profile: >> >> > Each sample counts as 0.01 seconds. >> >> > % cumulative self self total >> >> > time seconds seconds calls s/call s/call name >> >> > 34.00 31.13 31.13 72495 0.00 0.00 >> >> > Curl_hash_clean_with_criterium >> >> > 23.64 52.78 21.65 386070412 0.00 0.00 >> >> > hostcache_timestamp_remove >> >> > 13.28 64.94 12.16 1619440 0.00 0.00 Curl_hash_pick >> >> > 5.92 70.36 5.42 137653129 0.00 0.00 >> >> > Curl_str_key_compare >> >> > 5.50 75.40 5.04 203018 0.00 0.00 Curl_hash_add >> >> > 4.55 79.57 4.17 74618 0.00 0.00 create_conn >> >> > 1.70 81.13 1.56 2757148 0.00 0.00 dprintf_formatf >> >> > 0.98 82.03 0.90 76945 0.00 0.00 ConnectionStore >> >> > 0.95 82.90 0.87 455775100 0.00 0.00 >> >> > ares__is_list_empty >> >> > 0.57 83.42 0.53 >> >> > locking_function >> >> > 0.39 83.78 0.36 146742 0.00 0.00 ares_cancel >> >> > 0.35 84.10 0.32 102968806 0.00 0.00 addbyter >> >> > 0.35 84.42 0.32 456587 0.00 0.00 Curl_readwrite >> >> > 0.35 84.74 0.32 73133 0.00 0.00 >> >> > curl_multi_remove_handle >> >> > 0.34 85.05 0.31 1360545 0.00 0.00 multi_runsingle >> >> > >> >> > >> >> > 2012/3/6 Robert Iakobashvili <cor...@gm...> >> >> >> >> >> >> If it will not be helpful this means that recent >> >> >> changes in libcurl library have broken DNS interface. >> >> >> >> >> >> You can try a previos version and the latest libcurl, like: >> >> >> >> >> >> http://curl.haxx.se/download/curl-7.23.0.tar.gz >> >> >> and >> >> >> http://curl.haxx.se/downloadcurl-7.24.0.tar.gz >> >> >> >> >> >> by: >> >> >> >> >> >> 1. downloading a libcurl version and placing it to >> >> >> curl-loader-0.56/packages ; >> >> >> 2. correcting the Makefile in curl-loader-0.56 >> >> >> CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.24 or CURL_VER:=7.23 >> >> >> 3. make cleanall; make >> >> >> >> >> >> Please, send us your observations. >> >> >> Thanks! >> >> >> Robert >> >> >> >> >> >> On Mon, Mar 5, 2012 at 6:57 PM, Robert Iakobashvili >> >> >> <cor...@gm...> >> >> >> wrote: >> >> >> > If so, please try to change >> >> >> > >> >> >> > loader.c, line 578 which looks like: >> >> >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, -1); >> >> >> > >> >> >> > 1. instead of -1 try to set some value like 60 >> >> >> > 2. make clean; make >> >> >> > >> >> >> > Thanks, >> >> >> > Robert >> >> >> > >> >> >> > On Mon, Mar 5, 2012 at 6:19 PM, Fred Huang <di...@gm...> >> >> >> > wrote: >> >> >> >> >> >> >> >> Looks like the bottleneck comes from libcurl other than c-ares... >> >> >> >> test 3 (no ares patch applied): >> >> >> >> dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve >> >> >> >> *.com >> >> >> >> to one IP address >> >> >> >> number of domain names in URLs: 780,000 >> >> >> >> number of client: 1000 >> >> >> >> CPU usage: 85% >> >> >> >> cpu% irq% sirq% sys% iowt% mem_used buf&cached >> >> >> >> 85.6 0.0 0.8 5.9 0.0 3334.3Mb 112.0Mb >> >> >> >> SSL TPS: 320 >> >> >> >> SSL throughput: 200Mbps >> >> >> >> >> >> >> >> # gprof /usr/bin/curl-loader gmon.out -p | head -50 >> >> >> >> Flat profile: >> >> >> >> Each sample counts as 0.01 seconds. >> >> >> >> % cumulative self self total >> >> >> >> time seconds seconds calls s/call s/call name >> >> >> >> 40.80 120.16 120.16 116079 0.00 0.00 >> >> >> >> Curl_hash_clean_with_criterium >> >> >> >> 24.46 192.18 72.02 983166323 0.00 0.00 >> >> >> >> hostcache_timestamp_remove >> >> >> >> 13.30 231.35 39.17 2059341 0.00 0.00 >> >> >> >> Curl_hash_pick >> >> >> >> 6.45 250.35 19.00 240087 0.00 0.00 >> >> >> >> Curl_hash_add >> >> >> >> 4.81 264.51 14.16 334257102 0.00 0.00 >> >> >> >> Curl_str_key_compare >> >> >> >> 3.40 274.52 10.01 119514 0.00 0.00 create_conn >> >> >> >> 0.93 277.25 2.73 4478910 0.00 0.00 >> >> >> >> dprintf_formatf >> >> >> >> 0.90 279.91 2.66 122209 0.00 0.00 >> >> >> >> ConnectionStore >> >> >> >> 0.40 281.09 1.18 >> >> >> >> locking_function >> >> >> >> 0.27 281.88 0.79 114759 0.00 0.00 >> >> >> >> curl_multi_remove_handle >> >> >> >> 0.21 282.51 0.63 7993664 0.00 0.00 >> >> >> >> client_tracing_function >> >> >> >> 0.21 283.12 0.61 122685 0.00 0.00 >> >> >> >> Curl_num_addresses >> >> >> >> 0.19 283.68 0.56 397028 0.00 0.00 >> >> >> >> curl_multi_socket_action >> >> >> >> 0.19 284.23 0.55 2036116 0.00 0.00 >> >> >> >> multi_runsingle >> >> >> >> 0.17 284.72 0.49 703687 0.00 0.00 >> >> >> >> Curl_readwrite >> >> >> >> 0.16 285.19 0.47 id_function >> >> >> >> 0.15 285.62 0.43 29692029 0.00 0.00 >> >> >> >> Curl_socket_check >> >> >> >> 0.14 286.02 0.40 91973856 0.00 0.00 >> >> >> >> Curl_raw_toupper >> >> >> >> 0.13 286.40 0.38 164664144 0.00 0.00 addbyter >> >> >> >> 0.13 286.77 0.37 2180797 0.00 0.00 Curl_infof >> >> >> >> 0.11 287.10 0.33 122675 0.00 0.00 >> >> >> >> Curl_if_is_interface_name >> >> >> >> 0.10 287.39 0.29 471318 0.00 0.00 >> >> >> >> ossl_connect_common >> >> >> >> 0.10 287.67 0.28 2113528 0.00 0.00 >> >> >> >> Curl_raw_equal >> >> >> >> 0.09 287.93 0.26 8676 0.00 0.02 >> >> >> >> curl_multi_socket_all >> >> >> >> 0.08 288.17 0.24 1663262 0.00 0.00 Curl_splay >> >> >> >> 0.08 288.41 0.24 8505 0.00 0.01 >> >> >> >> curl_multi_perform >> >> >> >> 0.07 288.63 0.22 3114576 0.00 0.00 >> >> >> >> Curl_pgrsUpdate >> >> >> >> 0.06 288.80 0.17 119637 0.00 0.00 >> >> >> >> singleipconnect >> >> >> >> 0.06 288.97 0.17 7994818 0.00 0.00 Curl_debug >> >> >> >> 0.05 289.13 0.16 235820 0.00 0.00 event_del >> >> >> >> 0.05 289.29 0.16 119393071 0.00 0.00 >> >> >> >> curl_strequal >> >> >> >> 0.05 289.45 0.16 18535473 0.00 0.00 >> >> >> >> alloc_addbyter >> >> >> >> 0.05 289.61 0.16 356074 0.00 0.00 >> >> >> >> Curl_hash_str >> >> >> >> 0.05 289.76 0.15 1797233 0.00 0.00 >> >> >> >> fd_key_compare >> >> >> >> 0.05 289.91 0.15 234966 0.00 0.00 >> >> >> >> Curl_ssl_getsessionid >> >> >> >> 0.04 290.04 0.13 585853 0.00 0.00 >> >> >> >> Curl_llist_destroy >> >> >> >> 0.04 290.16 0.12 240826 0.00 0.00 Curl_resolv >> >> >> >> 0.03 290.26 0.10 7514328 0.00 0.00 >> >> >> >> Curl_raw_nequal >> >> >> >> 0.03 290.36 0.10 2214436 0.00 0.00 Curl_read >> >> >> >> 0.03 290.46 0.10 116872 0.00 0.00 >> >> >> >> Curl_http_readwrite_headers >> >> >> >> 0.03 290.56 0.10 >> >> >> >> ssl_tls_trace >> >> >> >> 0.03 290.66 0.10 8013097 0.00 0.00 >> >> >> >> scan_response >> >> >> >> 0.03 290.75 0.09 29488555 0.00 0.00 >> >> >> >> curlx_tvdiff >> >> >> >> 0.03 290.84 0.09 2142468 0.00 0.00 Curl_setopt >> >> >> >> 0.03 290.93 0.09 476299 0.00 0.00 Curl_expire >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> 2012/3/5 Robert Iakobashvili <cor...@gm...> >> >> >> >>> >> >> >> >>> Dear Fred Huang, >> >> >> >>> >> >> >> >>> Thanks for your posting. >> >> >> >>> >> >> >> >>> Could you please check if this is due to some badness at cares? >> >> >> >>> Thanks. >> >> >> >>> >> >> >> >>> Let's exclude usage of cares based DNS resolving by applying the >> >> >> >>> patch >> >> >> >>> attached >> >> >> >>> and use instead global DNS cache. >> >> >> >>> >> >> >> >>> cd ./curl-loader-0.56 >> >> >> >>> patch -p1 < path-to/disable-cares-use-global-dns.patch >> >> >> >>> make cleanall; make >> >> >> >>> >> >> >> >>> See if it helps. >> >> >> >>> >> >> >> >>> Global cache, unfortunately, is not thread safe. >> >> >> >>> >> >> >> >>> If it helps, we'll try to restore the previous version of cares. >> >> >> >>> >> >> >> >>> Thanks, >> >> >> >>> Robert >> >> >> >>> >> >> >> >>> On Mon, Mar 5, 2012 at 4:28 PM, Fred Huang <di...@gm...> >> >> >> >>> wrote: >> >> >> >>>> >> >> >> >>>> test 1: >> >> >> >>>> >> >> >> >>>> dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, >> >> >> >>>> resolve >> >> >> >>>> *.com to one IP address >> >> >> >>>> number of domain names in all https URLs: 780,000 >> >> >> >>>> number of client: 1000 >> >> >> >>>> CPU usage: 78% >> >> >> >>>> cpu% irq% sirq% sys% iowt% mem_used buf&cached >> >> >> >>>> 78.8 0.0 0.6 3.8 0.0 3640.9Mb 282.1Mb >> >> >> >>>> SSL TPS: 300 >> >> >> >>>> SSL throughput: 200Mbps >> >> >> >>>> >> >> >> >>>> # gprof /usr/bin/curl-loader gmon.out -p | head -50 >> >> >> >>>> Flat profile: >> >> >> >>>> Each sample counts as 0.01 seconds. >> >> >> >>>> % cumulative self self total >> >> >> >>>> time seconds seconds calls s/call s/call name >> >> >> >>>> 38.55 127.10 127.10 130975 0.00 0.00 >> >> >> >>>> Curl_hash_clean_with_criterium >> >> >> >>>> 24.63 208.32 81.22 1247528407 0.00 0.00 >> >> >> >>>> hostcache_timestamp_remove >> >> >> >>>> 13.08 251.44 43.12 2791988 0.00 0.00 >> >> >> >>>> Curl_hash_pick >> >> >> >>>> 7.30 275.51 24.07 352490 0.00 0.00 >> >> >> >>>> Curl_hash_add >> >> >> >>>> 6.01 295.31 19.80 410760770 0.00 0.00 >> >> >> >>>> Curl_str_key_compare >> >> >> >>>> 2.95 305.04 9.73 131015 0.00 0.00 >> >> >> >>>> create_conn >> >> >> >>>> 0.88 307.94 2.90 4841390 0.00 0.00 >> >> >> >>>> dprintf_formatf >> >> >> >>>> 0.74 310.38 2.44 136036 0.00 0.00 >> >> >> >>>> ConnectionStore >> >> >> >>>> 0.48 311.96 1.58 814604911 0.00 0.00 >> >> >> >>>> ares__is_list_empty >> >> >> >>>> 0.36 313.16 1.20 >> >> >> >>>> locking_function >> >> >> >>>> 0.27 314.06 0.90 261608 0.00 0.00 >> >> >> >>>> ares_cancel >> >> >> >>>> 0.22 314.79 0.73 127417 0.00 0.00 >> >> >> >>>> curl_multi_remove_handle >> >> >> >>>> 0.19 315.40 0.62 8906301 0.00 0.00 >> >> >> >>>> client_tracing_function >> >> >> >>>> 0.17 315.97 0.57 135305 0.00 0.00 >> >> >> >>>> Curl_num_addresses >> >> >> >>>> 0.17 316.54 0.57 181937907 0.00 0.00 addbyter >> >> >> >>>> 0.17 317.09 0.55 2374363 0.00 0.00 >> >> >> >>>> multi_runsingle >> >> >> >>>> 0.16 317.62 0.53 541954 0.00 0.00 >> >> >> >>>> ares__init_list_node >> >> >> >>>> 0.16 318.14 0.52 494412 0.00 0.00 >> >> >> >>>> curl_multi_socket_action >> >> >> >>>> 0.14 318.60 0.46 33578897 0.00 0.00 >> >> >> >>>> Curl_socket_check >> >> >> >>>> 0.13 319.04 0.44 102951052 0.00 0.00 >> >> >> >>>> Curl_raw_toupper >> >> >> >>>> 0.13 319.48 0.44 787558 0.00 0.00 >> >> >> >>>> Curl_readwrite >> >> >> >>>> 0.13 319.91 0.44 >> >> >> >>>> id_function >> >> >> >>>> 0.12 320.29 0.38 391390 0.00 0.00 >> >> >> >>>> Curl_hash_str >> >> >> >>>> 0.09 320.58 0.29 9216 0.00 0.00 >> >> >> >>>> curl_multi_perform >> >> >> >>>> 0.08 320.86 0.28 2350908 0.00 0.00 >> >> >> >>>> Curl_raw_equal >> >> >> >>>> 0.08 321.14 0.28 2019415 0.00 0.00 >> >> >> >>>> Curl_splay >> >> >> >>>> 0.08 321.40 0.26 520625 0.00 0.00 >> >> >> >>>> ossl_connect_common >> >> >> >>>> 0.08 321.65 0.25 3532992 0.00 0.00 >> >> >> >>>> Curl_pgrsUpdate >> >> >> >>>> 0.07 321.87 0.22 131778537 0.00 0.00 >> >> >> >>>> curl_strequal >> >> >> >>>> 0.05 322.05 0.18 8954118 0.00 0.00 >> >> >> >>>> scan_response >> >> >> >>>> 0.05 322.23 0.18 533644 0.00 0.00 >> >> >> >>>> ares_expand_name >> >> >> >>>> 0.05 322.40 0.17 2477345 0.00 0.00 >> >> >> >>>> fd_key_compare >> >> >> >>>> 0.05 322.57 0.17 2429135 0.00 0.00 >> >> >> >>>> Curl_infof >> >> >> >>>> 0.05 322.74 0.17 130966 0.00 0.00 >> >> >> >>>> singleipconnect >> >> >> >>>> 0.05 322.90 0.16 9292 0.00 0.00 >> >> >> >>>> curl_multi_socket_all >> >> >> >>>> 0.05 323.06 0.16 270137 0.00 0.00 >> >> >> >>>> ares__get_hostent >> >> >> >>>> 0.05 323.21 0.15 8943657 0.00 0.00 >> >> >> >>>> Curl_debug >> >> >> >>>> 0.04 323.35 0.14 262493 0.00 0.00 >> >> >> >>>> Curl_ssl_getsessionid >> >> >> >>>> 0.04 323.48 0.13 566006 0.00 0.00 >> >> >> >>>> socket_callback >> >> >> >>>> 0.04 323.61 0.13 33983864 0.00 0.00 >> >> >> >>>> curlx_tvdiff >> >> >> >>>> 0.04 323.74 0.13 132853 0.00 0.00 >> >> >> >>>> Curl_http_readwrite_headers >> >> >> >>>> 0.04 323.86 0.12 344032 0.00 0.00 epoll_del >> >> >> >>>> 0.04 323.98 0.12 223477 0.00 0.00 Curl_poll >> >> >> >>>> 0.04 324.10 0.12 8526280 0.00 0.00 >> >> >> >>>> Curl_raw_nequal >> >> >> >>>> 0.03 324.21 0.11 348839 0.00 0.00 event_del >> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >> >>>> test 2: >> >> >> >>>> >> >> >> >>>> dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, >> >> >> >>>> resolve >> >> >> >>>> *.com to one IP address >> >> >> >>>> number of domain names in all https URLs: 1 >> >> >> >>>> number of client: 1000 >> >> >> >>>> CPU usage: 75% >> >> >> >>>> SSL TPS: 1300 >> >> >> >>>> SSL throughput: 700Mbps >> >> >> >>>> >> >> >> >>>> # gprof /usr/bin/curl-loader gmon.out -p | head -50 >> >> >> >>>> Flat profile: >> >> >> >>>> Each sample counts as 0.01 seconds. >> >> >> >>>> % cumulative self self total >> >> >> >>>> time seconds seconds calls s/call s/call name >> >> >> >>>> 8.98 2.45 2.45 5127312 0.00 0.00 >> >> >> >>>> dprintf_formatf >> >> >> >>>> 8.90 4.88 2.43 1292796503 0.00 0.00 >> >> >> >>>> ares__is_list_empty >> >> >> >>>> 6.12 6.55 1.67 322461 0.00 0.00 >> >> >> >>>> create_conn >> >> >> >>>> 4.58 7.80 1.25 419872 0.00 0.00 >> >> >> >>>> ares_cancel >> >> >> >>>> 4.10 8.92 1.12 2100162 0.00 0.00 >> >> >> >>>> Curl_readwrite >> >> >> >>>> 3.92 9.99 1.07 4527388 0.00 0.00 >> >> >> >>>> Curl_hash_pick >> >> >> >>>> 3.43 10.93 0.94 >> >> >> >>>> locking_function >> >> >> >>>> 3.04 11.76 0.83 909759 0.00 0.00 >> >> >> >>>> curl_multi_socket_action >> >> >> >>>> 3.00 12.58 0.82 97252 0.00 0.00 >> >> >> >>>> Curl_hash_clean_with_criterium >> >> >> >>>> 2.71 13.32 0.74 179704016 0.00 0.00 >> >> >> >>>> Curl_raw_toupper >> >> >> >>>> 2.68 14.05 0.73 4646524 0.00 0.00 >> >> >> >>>> multi_runsingle >> >> >> >>>> 2.44 14.71 0.67 13862992 0.00 0.00 >> >> >> >>>> client_tracing_function >> >> >> >>>> 2.42 15.37 0.66 319735 0.00 0.00 >> >> >> >>>> curl_multi_remove_handle >> >> >> >>>> 2.42 16.03 0.66 22 0.03 0.03 >> >> >> >>>> ares__init_list_node >> >> >> >>>> 1.94 16.56 0.53 9930244 0.00 0.00 >> >> >> >>>> hostcache_timestamp_remove >> >> >> >>>> 1.94 17.09 0.53 169320088 0.00 0.00 addbyter >> >> >> >>>> 1.80 17.58 0.49 >> >> >> >>>> id_function >> >> >> >>>> 1.36 17.95 0.37 7057146 0.00 0.00 >> >> >> >>>> Curl_pgrsUpdate >> >> >> >>>> 1.36 18.32 0.37 16835 0.00 0.00 >> >> >> >>>> curl_multi_socket_all >> >> >> >>>> 1.25 18.66 0.34 16719 0.00 0.00 >> >> >> >>>> curl_multi_perform >> >> >> >>>> 1.21 18.99 0.33 329403 0.00 0.00 >> >> >> >>>> Curl_http_readwrite_headers >> >> >> >>>> 1.17 19.31 0.32 4036624 0.00 0.00 >> >> >> >>>> Curl_splay >> >> >> >>>> 1.14 19.62 0.31 20838902 0.00 0.00 >> >> >> >>>> Curl_raw_nequal >> >> >> >>>> 0.82 19.85 0.23 2764123 0.00 0.00 >> >> >> >>>> Curl_infof >> >> >> >>>> 0.81 20.07 0.22 5933857 0.00 0.00 ossl_recv >> >> >> >>>> 0.77 20.28 0.21 323112 0.00 0.00 >> >> >> >>>> Curl_splayremovebyaddr >> >> >> >>>> 0.75 20.48 0.21 5943991 0.00 0.00 Curl_read >> >> >> >>>> 0.70 20.67 0.19 425174 0.00 0.00 event_del >> >> >> >>>> 0.70 20.86 0.19 98652 0.00 0.00 >> >> >> >>>> Curl_if_is_interface_name >> >> >> >>>> 0.70 21.05 0.19 8677733 0.00 0.00 >> >> >> >>>> Curl_socket_check >> >> >> >>>> 0.59 21.21 0.16 4129554 0.00 0.00 >> >> >> >>>> fd_key_compare >> >> >> >>>> 0.57 21.37 0.16 14017019 0.00 0.00 >> >> >> >>>> Curl_debug >> >> >> >>>> 0.55 21.52 0.15 9376861 0.00 0.00 >> >> >> >>>> stat_data_in_add >> >> >> >>>> 0.55 21.67 0.15 6840468 0.00 0.00 >> >> >> >>>> Curl_timeleft >> >> >> >>>> 0.55 21.82 0.15 2054602 0.00 0.00 >> >> >> >>>> Curl_raw_equal >> >> >> >>>> 0.55 21.97 0.15 1291611 0.00 0.00 >> >> >> >>>> Curl_expire >> >> >> >>>> 0.51 22.11 0.14 16570063 0.00 0.00 >> >> >> >>>> curlx_tvnow >> >> >> >>>> 0.51 22.25 0.14 13863284 0.00 0.00 >> >> >> >>>> scan_response >> >> >> >>>> 0.51 22.39 0.14 5498864 0.00 0.00 >> >> >> >>>> Curl_setopt >> >> >> >>>> 0.48 22.52 0.13 97490291 0.00 0.00 >> >> >> >>>> curl_strequal >> >> >> >>>> 0.44 22.64 0.12 324253 0.00 0.00 Curl_http >> >> >> >>>> 0.44 22.76 0.12 394386 0.00 0.00 >> >> >> >>>> ossl_connect_common >> >> >> >>>> 0.40 22.87 0.11 3581335 0.00 0.00 >> >> >> >>>> Curl_getinfo >> >> >> >>>> 0.38 22.97 0.11 26983092 0.00 0.00 >> >> >> >>>> alloc_addbyter >> >> >> >>>> 0.37 23.07 0.10 8729044 0.00 0.00 >> >> >> >>>> Curl_client_write >> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >> >>>> ------------------------------------------------------------------------------ >> >> >> >>>> Try before you buy = See our experts in action! >> >> >> >>>> The most comprehensive online learning library for Microsoft >> >> >> >>>> developers >> >> >> >>>> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, >> >> >> >>>> CSS3, >> >> >> >>>> MVC3, >> >> >> >>>> Metro Style Apps, more. Free future releases when you subscribe >> >> >> >>>> now! >> >> >> >>>> http://p.sf.net/sfu/learndevnow-dev2 >> >> >> >>>> _______________________________________________ >> >> >> >>>> curl-loader-devel mailing list >> >> >> >>>> cur...@li... >> >> >> >>>> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel >> >> >> >>>> >> >> >> >>> >> >> >> >>> >> >> >> >>> >> >> >> >>> -- >> >> >> >>> Regards, >> >> >> >>> Robert Iakobashvili, Ph.D. >> >> >> >>> >> >> >> >>> Home: http://www.ghotit.com >> >> >> >>> Blog: http://www.ghotit.com/category/ghotit-blogs/ >> >> >> >>> Twitter: http://twitter.com/ghotit >> >> >> >>> Facebook: http://facebook.com/ghotit >> >> >> >>> >> >> >> >>> >> >> >> >>> ...................................................................... >> >> >> >>> Ghotit Dyslexia >> >> >> >>> Assistive technology that understands you >> >> >> >>> >> >> >> >>> >> >> >> >>> ...................................................................... >> >> >> >>> >> >> >> >>> >> >> >> >>> >> >> >> >>> >> >> >> >>> >> >> >> >>> ------------------------------------------------------------------------------ >> >> >> >>> Try before you buy = See our experts in action! >> >> >> >>> The most comprehensive online learning library for Microsoft >> >> >> >>> developers >> >> >> >>> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, >> >> >> >>> CSS3, >> >> >> >>> MVC3, >> >> >> >>> Metro Style Apps, more. Free future releases when you subscribe >> >> >> >>> now! >> >> >> >>> http://p.sf.net/sfu/learndevnow-dev2 >> >> >> >>> _______________________________________________ >> >> >> >>> curl-loader-devel mailing list >> >> >> >>> cur...@li... >> >> >> >>> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel >> >> >> >>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> ------------------------------------------------------------------------------ >> >> >> >> Try before you buy = See our experts in action! >> >> >> >> The most comprehensive online learning library for Microsoft >> >> >> >> developers >> >> >> >> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, >> >> >> >> CSS3, >> >> >> >> MVC3, >> >> >> >> Metro Style Apps, more. Free future releases when you subscribe >> >> >> >> now! >> >> >> >> http://p.sf.net/sfu/learndevnow-dev2 >> >> >> >> _______________________________________________ >> >> >> >> curl-loader-devel mailing list >> >> >> >> cur...@li... >> >> >> >> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel >> >> >> >> >> >> >> > >> >> >> > >> >> >> > >> >> >> > -- >> >> >> > Regards, >> >> >> > Robert Iakobashvili, Ph.D. >> >> >> > >> >> >> > Home: http://www.ghotit.com >> >> >> > Blog: http://www.ghotit.com/category/ghotit-blogs/ >> >> >> > Twitter: http://twitter.com/ghotit >> >> >> > Facebook: http://facebook.com/ghotit >> >> >> > >> >> >> > >> >> >> > ...................................................................... >> >> >> > Ghotit Dyslexia >> >> >> > Assistive technology that understands you >> >> >> > >> >> >> > >> >> >> > ...................................................................... >> >> >> >> >> >> >> >> >> >> >> >> -- >> >> >> Regards, >> >> >> Robert Iakobashvili, Ph.D. >> >> >> >> >> >> Home: http://www.ghotit.com >> >> >> Blog: http://www.ghotit.com/category/ghotit-blogs/ >> >> >> Twitter: http://twitter.com/ghotit >> >> >> Facebook: http://facebook.com/ghotit >> >> >> >> >> >> ...................................................................... >> >> >> Ghotit Dyslexia >> >> >> Assistive technology that understands you >> >> >> >> >> >> ...................................................................... >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> ------------------------------------------------------------------------------ >> >> >> Try before you buy = See our experts in action! >> >> >> The most comprehensive online learning library for Microsoft >> >> >> developers >> >> >> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, >> >> >> MVC3, >> >> >> Metro Style Apps, more. Free future releases when you subscribe now! >> >> >> http://p.sf.net/sfu/learndevnow-dev2 >> >> >> _______________________________________________ >> >> >> curl-loader-devel mailing list >> >> >> cur...@li... >> >> >> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > ------------------------------------------------------------------------------ >> >> > Keep Your Developer Skills Current with LearnDevNow! >> >> > The most comprehensive online learning library for Microsoft >> >> > developers >> >> > is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, >> >> > MVC3, >> >> > Metro Style Apps, more. Free future releases when you subscribe now! >> >> > http://p.sf.net/sfu/learndevnow-d2d >> >> > _______________________________________________ >> >> > curl-loader-devel mailing list >> >> > cur...@li... >> >> > https://lists.sourceforge.net/lists/listinfo/curl-loader-devel >> >> > >> >> >> >> >> >> >> >> -- >> >> Regards, >> >> Robert Iakobashvili, Ph.D. >> >> >> >> Home: http://www.ghotit.com >> >> Blog: http://www.ghotit.com/category/ghotit-blogs/ >> >> Twitter: http://twitter.com/ghotit >> >> Facebook: http://facebook.com/ghotit >> >> ...................................................................... >> >> Ghotit Dyslexia >> >> Assistive technology that understands you >> >> ...................................................................... >> >> >> >> >> >> >> >> ------------------------------------------------------------------------------ >> >> Keep Your Developer Skills Current with LearnDevNow! >> >> The most comprehensive online learning library for Microsoft developers >> >> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, >> >> MVC3, >> >> Metro Style Apps, more. Free future releases when you subscribe now! >> >> http://p.sf.net/sfu/learndevnow-d2d >> >> _______________________________________________ >> >> curl-loader-devel mailing list >> >> cur...@li... >> >> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel >> > >> > >> > >> > >> > ------------------------------------------------------------------------------ >> > Keep Your Developer Skills Current with LearnDevNow! >> > The most comprehensive online learning library for Microsoft developers >> > is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, >> > Metro Style Apps, more. Free future releases when you subscribe now! >> > http://p.sf.net/sfu/learndevnow-d2d >> > _______________________________________________ >> > curl-loader-devel mailing list >> > cur...@li... >> > https://lists.sourceforge.net/lists/listinfo/curl-loader-devel >> > >> >> >> >> -- >> Regards, >> Robert Iakobashvili, Ph.D. >> >> Home: http://www.ghotit.com >> Blog: http://www.ghotit.com/category/ghotit-blogs/ >> Twitter: http://twitter.com/ghotit >> Facebook: http://facebook.com/ghotit >> ...................................................................... >> Ghotit Dyslexia >> Assistive technology that understands you >> ...................................................................... >> >> >> ------------------------------------------------------------------------------ >> Keep Your Developer Skills Current with LearnDevNow! >> The most comprehensive online learning library for Microsoft developers >> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, >> Metro Style Apps, more. Free future releases when you subscribe now! >> http://p.sf.net/sfu/learndevnow-d2d >> _______________________________________________ >> curl-loader-devel mailing list >> cur...@li... >> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel > > > > ------------------------------------------------------------------------------ > Virtualization & Cloud Management Using Capacity Planning > Cloud computing makes use of virtualization - but cloud computing > also focuses on allowing computing to be delivered as a service. > http://www.accelacomm.com/jaw/sfnl/114/51521223/ > _______________________________________________ > curl-loader-devel mailing list > cur...@li... > https://lists.sourceforge.net/lists/listinfo/curl-loader-devel > -- Regards, Robert Iakobashvili, Ph.D. Home: http://www.ghotit.com Blog: http://www.ghotit.com/category/ghotit-blogs/ Twitter: http://twitter.com/ghotit Facebook: http://facebook.com/ghotit ...................................................................... Ghotit Dyslexia Assistive technology that understands you ...................................................................... |
|
From: Fred H. <di...@gm...> - 2012-03-07 00:44:00
|
yes, I use -t 8 option because the test box has 8 logic CPUs (Dell 990, Intel Core i7 2600, 16GB memory). if I do not use -t option, the CPU used by curl-loader will be 100% on one of 8 logic CPU (12% of total CPU), and the CAPS drops to 100: # gprof /usr/bin/curl-loader gmon.out -p | head -20 Flat profile: Each sample counts as 0.01 seconds. % cumulative self self total time seconds seconds calls s/call s/call name 34.12 2.74 2.74 17385 0.00 0.00 create_conn 11.21 3.64 0.90 30998583 0.00 0.00 hostcache_timestamp_remove 9.84 4.43 0.79 13433 0.00 0.00 Curl_hash_clean_with_criterium 7.97 5.07 0.64 381818 0.00 0.00 Curl_hash_pick 5.85 5.54 0.47 17385 0.00 0.00 ConnectionStore 5.11 5.95 0.41 16385 0.00 0.00 curl_multi_remove_handle 4.11 6.28 0.33 16509889 0.00 0.00 Curl_str_key_compare 3.36 6.55 0.27 23959914 0.00 0.00 Curl_socket_check 3.24 6.81 0.26 56703 0.00 0.00 Curl_hash_add 2.62 7.02 0.21 635750 0.00 0.00 dprintf_formatf 1.25 7.12 0.10 92421190 0.00 0.00 ares__is_list_empty 0.75 7.18 0.06 97957 0.00 0.00 Curl_readwrite 0.62 7.23 0.05 23422260 0.00 0.00 addbyter 0.62 7.28 0.05 1136775 0.00 0.00 client_tracing_function 0.50 7.32 0.04 3091375 0.00 0.00 ares__init_list_head I also tried to use *curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, 20);* but get no improvement on CAPS. Regards, Fred 2012/3/7 Robert Iakobashvili <cor...@gm...> > Dear Fred, > > If I understand correctly your observations, the issue is not in: > > 1. DNS resolving; > 2. Version of cares or libcurl; > > but rather in > > Less effective handling of HTTPS versus HTTP in libcurl. > > Please, confirm that my understanding is correct. > > > HTTPS URLs are using an extra locking in openssl with -t option > that could be making the issue. > If you are using -t option, try to load without it and let's say with > 300 clients. > > If you are not using -t, thus the only way to proceed is to share > your observations with libcurl people at > > libcurl development <cur...@co...> > > Thank you. > > Best wishes. > Robert > > On Tue, Mar 6, 2012 at 6:17 PM, Fred Huang <di...@gm...> wrote: > > 1. the major CPU consumer is curl-loader. here is the top: > > top - 07:36:36 up 23:31, 5 users, load average: 4.32, 2.26, 0.91 > > Tasks: 237 total, 2 running, 235 sleeping, 0 stopped, 0 zombie > > Cpu0 : 76.6%us, 3.6%sy, 0.0%ni, 19.4%id, 0.0%wa, 0.0%hi, 0.3%si, > 0.0% > > Cpu1 : 72.4%us, 4.0%sy, 0.0%ni, 22.9%id, 0.0%wa, 0.0%hi, 0.7%si, > 0.0% > > Cpu2 : 65.8%us, 3.3%sy, 0.0%ni, 30.6%id, 0.0%wa, 0.0%hi, 0.3%si, > 0.0% > > Cpu3 : 75.2%us, 4.3%sy, 0.0%ni, 19.5%id, 0.0%wa, 0.0%hi, 1.0%si, > 0.0% > > Cpu4 : 79.1%us, 4.0%sy, 0.0%ni, 16.8%id, 0.0%wa, 0.0%hi, 0.0%si, > 0.0% > > Cpu5 : 75.8%us, 4.3%sy, 0.0%ni, 19.2%id, 0.0%wa, 0.0%hi, 0.7%si, > 0.0% > > Cpu6 : 74.1%us, 5.0%sy, 0.0%ni, 20.3%id, 0.0%wa, 0.0%hi, 0.7%si, > 0.0% > > Cpu7 : 78.7%us, 4.0%sy, 0.0%ni, 16.6%id, 0.0%wa, 0.0%hi, 0.7%si, > 0.0% > > Mem: 16209932k total, 3829500k used, 12380432k free, 99112k buffers > > Swap: 18448376k total, 0k used, 18448376k free, 320284k cached > > > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > > 30795 root 20 0 3200m 2.8g 2296 S 79.2 18.4 21:26.84 curl-loader > > 1832 nobody 20 0 13944 1736 580 S 0.1 0.0 0:32.65 dnsmasq > > 9 root 20 0 0 0 0 S 0.0 0.0 0:00.53 ksoftirqd/1 > > 30825 root 20 0 15220 1360 960 R 0.0 0.0 0:00.42 top > > 1 root 20 0 19400 1568 1256 S 0.0 0.0 0:00.75 init > > 2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd > > 3 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/0 > > 4 root 20 0 0 0 0 S 0.0 0.0 0:00.06 ksoftirqd/0 > > 5 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/0 > > > > > > 2. 1000 loading clients > > > > 3. yes, tried 0.52 but no lucky -- CAPS 380, throughput 200Mbps, CPU 90% > > top - 08:17:10 up 1 day, 12 min, 8 users, load average: 4.66, 3.39, > 1.88 > > Tasks: 245 total, 2 running, 243 sleeping, 0 stopped, 0 zombie > > Cpu0 : 82.2%us, 4.0%sy, 0.0%ni, 13.9%id, 0.0%wa, 0.0%hi, 0.0%si, > 0.0% > > Cpu1 : 86.1%us, 4.3%sy, 0.0%ni, 7.9%id, 0.0%wa, 0.0%hi, 1.7%si, > 0.0% > > Cpu2 : 85.3%us, 4.0%sy, 0.0%ni, 10.0%id, 0.0%wa, 0.0%hi, 0.7%si, > 0.0% > > Cpu3 : 81.8%us, 3.0%sy, 0.0%ni, 14.2%id, 0.0%wa, 0.0%hi, 1.0%si, > 0.0% > > Cpu4 : 83.7%us, 4.0%sy, 0.0%ni, 12.3%id, 0.0%wa, 0.0%hi, 0.0%si, > 0.0% > > Cpu5 : 92.7%us, 3.3%sy, 0.0%ni, 3.3%id, 0.0%wa, 0.0%hi, 0.7%si, > 0.0% > > Cpu6 : 89.7%us, 2.0%sy, 0.0%ni, 8.0%id, 0.0%wa, 0.0%hi, 0.3%si, > 0.0% > > Cpu7 : 88.1%us, 3.6%sy, 0.0%ni, 6.6%id, 0.0%wa, 0.0%hi, 1.7%si, > 0.0% > > Mem: 16209932k total, 1795748k used, 14414184k free, 102824k buffers > > Swap: 18448376k total, 0k used, 18448376k free, 382988k cached > > > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > > 12236 root 20 0 1150m 894m 2572 S 90.7 5.7 51:11.91 curl-loader > > 1832 nobody 20 0 13944 1736 580 S 0.1 0.0 0:47.04 dnsmasq > > > > ]# gprof /usr/bin/curl-loader gmon.out -p | head -20 > > Flat profile: > > Each sample counts as 0.01 seconds. > > % cumulative self self total > > time seconds seconds calls s/call s/call name > > 43.92 151.95 151.95 157295 0.00 0.00 > > Curl_hash_clean_with_criterium > > 22.90 231.17 79.22 1712190971 0.00 0.00 > > hostcache_timestamp_remove > > 13.15 276.68 45.51 3260321 0.00 0.00 Curl_hash_pick > > 7.51 302.65 25.97 394171 0.00 0.00 Curl_hash_add > > 5.98 323.34 20.68 567437518 0.00 0.00 > Curl_str_key_compare > > 1.00 326.80 3.46 5380220 0.00 0.00 dprintf_formatf > > 0.41 328.21 1.41 locking_function > > 0.32 329.32 1.11 469222138 0.00 0.00 > ares__is_list_empty > > 0.31 330.40 1.08 3284508 0.00 0.00 multi_runsingle > > 0.28 331.38 0.98 10006806 0.00 0.00 > > client_tracing_function > > 0.24 332.21 0.83 151951 0.00 0.00 Curl_num_addresses > > 0.22 332.97 0.77 188075672 0.00 0.00 addbyter > > 0.19 333.64 0.67 841840 0.00 0.00 Curl_readwrite > > 0.19 334.29 0.65 id_function > > 0.18 334.90 0.61 39843 0.00 0.01 > curl_multi_socket_all > > > > > > > > 4. yes, it is correct for 0.56. If use same URL list by replacing https > to > > http, I can get CAPS 1500 and throughput 870Mbps with CPU% < 60% > > > > > > 2012/3/6 Robert Iakobashvili <cor...@gm...> > >> > >> Several questions: > >> > >> 1. Could you please run top and see if there are other processes burning > >> CPU? > >> 2. What is the number of loading clients that you use? > >> 3. Could you try curl-loader-0.52 if it compiles for you? > >> If not, please try to place curl-loader-7.19.7.tar.gz and > >> c-ares-16.0.tar.gz, > >> update their versions in Makefile and rebuild by make cleanall; make > >> > >> 4. Is it correct to say that for HTTP URLs (not HTTPS) you are getting > >> a good throughput > >> and CAPS with curl-loader-0.56? > >> > >> Thanks, > >> Robert > >> > >> On Tue, Mar 6, 2012 at 3:52 AM, Fred Huang <di...@gm...> wrote: > >> > TIMEOUT -1 -> 60 works a little better but no performance improvement > on > >> > CAPS or throughput. > >> > > >> > test 5: > >> > enable c-ares; > >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, 60); > >> > > >> > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve > *.com > >> > to > >> > one IP address > >> > number of domain names in URLs: 780,000 > >> > number of client: 1000 > >> > CPU usage: 74% > >> > cpu% irq% sirq% sys% iowt% mem_used buf&cached > >> > 74.0 0.0 0.7 4.2 0.0 3530.6Mb 250.8Mb > >> > SSL TPS: 360 > >> > SSL throughput: 220Mbps > >> > > >> > # gprof /usr/bin/curl-loader gmon.out -p | head -50 > >> > Flat profile: > >> > Each sample counts as 0.01 seconds. > >> > % cumulative self self total > >> > time seconds seconds calls s/call s/call name > >> > 26.63 23.13 23.13 119226 0.00 0.00 > >> > Curl_hash_clean_with_criterium > >> > 20.34 40.80 17.67 317792146 0.00 0.00 > >> > hostcache_timestamp_remove > >> > 11.27 50.59 9.79 2579903 0.00 0.00 Curl_hash_pick > >> > 7.84 57.40 6.81 120606 0.00 0.00 create_conn > >> > 5.65 62.31 4.91 119117401 0.00 0.00 > >> > Curl_str_key_compare > >> > 4.98 66.64 4.33 329829 0.00 0.00 Curl_hash_add > >> > 2.80 69.08 2.44 4476606 0.00 0.00 dprintf_formatf > >> > 2.57 71.31 2.23 124019 0.00 0.00 ConnectionStore > >> > 1.50 72.61 1.30 740997609 0.00 0.00 > >> > ares__is_list_empty > >> > 1.16 73.61 1.01 locking_function > >> > 0.76 74.27 0.66 238744 0.00 0.00 ares_cancel > >> > 0.70 74.88 0.61 457564 0.00 0.00 > >> > curl_multi_socket_action > >> > 0.67 75.46 0.58 118616 0.00 0.00 > >> > curl_multi_remove_handle > >> > 0.58 75.97 0.51 8211502 0.00 0.00 > >> > client_tracing_function > >> > 0.53 76.43 0.46 733561 0.00 0.00 Curl_readwrite > >> > 0.51 76.87 0.44 2183179 0.00 0.00 multi_runsingle > >> > 0.51 77.31 0.44 166585567 0.00 0.00 addbyter > >> > 0.44 77.69 0.38 2175805 0.00 0.00 Curl_raw_equal > >> > 0.43 78.06 0.38 id_function > >> > 0.40 78.41 0.35 29789346 0.00 0.00 > Curl_socket_check > >> > 0.40 78.76 0.35 479671 0.00 0.00 > >> > ossl_connect_common > >> > 0.40 79.11 0.35 490639 0.00 0.00 > >> > ares__init_list_node > >> > 0.33 79.40 0.29 94197670 0.00 0.00 Curl_raw_toupper > >> > 0.33 79.69 0.29 3268031 0.00 0.00 Curl_pgrsUpdate > >> > 0.28 79.93 0.24 8797 0.00 0.00 > curl_multi_perform > >> > 0.27 80.17 0.24 2233641 0.00 0.00 Curl_infof > >> > 0.26 80.40 0.23 1839837 0.00 0.00 Curl_splay > >> > 0.24 80.61 0.21 121713028 0.00 0.00 curl_strequal > >> > 0.21 80.79 0.18 243333 0.00 0.00 > >> > Curl_ssl_getsessionid > >> > 0.18 80.95 0.16 122180 0.00 0.00 > Curl_num_addresses > >> > 0.17 81.09 0.15 8271666 0.00 0.00 Curl_debug > >> > 0.16 81.23 0.14 18902452 0.00 0.00 alloc_addbyter > >> > 0.16 81.37 0.14 8851 0.00 0.00 > >> > curl_multi_socket_all > >> > 0.14 81.50 0.13 2236769 0.00 0.00 Curl_read > >> > 0.14 81.62 0.12 2271700 0.00 0.00 fd_key_compare > >> > 0.14 81.74 0.12 323167 0.00 0.00 event_del > >> > 0.14 81.86 0.12 205436 0.00 0.00 read_udp_packets > >> > 0.14 81.98 0.12 200979 0.00 0.00 processfds > >> > 0.13 82.09 0.11 123886 0.00 0.00 > >> > Curl_if_is_interface_name > >> > 0.13 82.20 0.11 120350 0.00 0.00 Curl_http > >> > 0.13 82.31 0.11 ssl_tls_trace > >> > 0.12 82.41 0.10 119229 0.00 0.00 > >> > Curl_splayremovebyaddr > >> > 0.12 82.51 0.10 7798717 0.00 0.00 Curl_raw_nequal > >> > 0.12 82.61 0.10 784485 0.00 0.00 event_add > >> > 0.10 82.70 0.09 643482 0.00 0.00 Curl_expire > >> > > >> > > >> > > >> > > >> > > >> > test 6: > >> > enable c-ares; > >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, 60); > >> > CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.23.0 > >> > > >> > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve > *.com > >> > to > >> > one IP address > >> > number of domain names in URLs: 780,000 > >> > number of client: 1000 > >> > CPU usage: 86% > >> > cpu% irq% sirq% sys% iowt% mem_used buf&cached > >> > 85.9 0.0 0.2 2.1 0.0 3537.2Mb 261.4Mb > >> > SSL TPS: 360 > >> > SSL throughput: 200Mbps > >> > > >> > # gprof /usr/bin/curl-loader gmon.out -p | head -20 > >> > Flat profile: > >> > Each sample counts as 0.01 seconds. > >> > % cumulative self self total > >> > time seconds seconds calls s/call s/call name > >> > 27.19 19.96 19.96 103472 0.00 0.00 > >> > Curl_hash_clean_with_criterium > >> > 21.73 35.91 15.95 274168981 0.00 0.00 > >> > hostcache_timestamp_remove > >> > 10.33 43.49 7.58 2256441 0.00 0.00 Curl_hash_pick > >> > 7.21 48.78 5.29 104540 0.00 0.00 create_conn > >> > 6.20 53.33 4.55 103413405 0.00 0.00 > >> > Curl_str_key_compare > >> > 4.88 56.91 3.58 287109 0.00 0.00 Curl_hash_add > >> > 3.02 59.13 2.22 3797855 0.00 0.00 dprintf_formatf > >> > 2.10 60.67 1.54 107711 0.00 0.00 ConnectionStore > >> > 1.65 61.88 1.21 642725327 0.00 0.00 > >> > ares__is_list_empty > >> > 1.19 62.76 0.88 locking_function > >> > 0.87 63.40 0.64 207817 0.00 0.00 ares_cancel > >> > 0.56 63.81 0.41 404108 0.00 0.00 > >> > curl_multi_socket_action > >> > 0.54 64.21 0.40 425603 0.00 0.00 > >> > ares__init_list_node > >> > 0.53 64.60 0.39 7068112 0.00 0.00 > >> > client_tracing_function > >> > 0.49 64.96 0.36 1908587 0.00 0.00 multi_runsingle > >> > > >> > > >> > > >> > test 7: > >> > enable c-ares; > >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, -1); > >> > CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.23.0 > >> > > >> > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve > *.com > >> > to > >> > one IP address > >> > number of domain names in URLs: 780,000 > >> > number of client: 1000 > >> > CPU usage: 86% > >> > cpu% irq% sirq% sys% iowt% mem_used buf&cached > >> > 86.6 0.0 0.6 3.3 0.0 3624.8Mb 263.0Mb > >> > SSL TPS: 300 > >> > SSL throughput: 200Mbps > >> > > >> > # gprof /usr/bin/curl-loader gmon.out -p | head -20 > >> > Flat profile: > >> > Each sample counts as 0.01 seconds. > >> > % cumulative self self total > >> > time seconds seconds calls s/call s/call name > >> > 39.83 369.71 369.71 207173 0.00 0.00 > >> > Curl_hash_clean_with_criterium > >> > 24.27 594.98 225.27 3198337733 0.00 0.00 > >> > hostcache_timestamp_remove > >> > 13.75 722.59 127.61 4312864 0.00 0.00 Curl_hash_pick > >> > 9.19 807.87 85.28 546507 0.00 0.00 Curl_hash_add > >> > 6.14 864.88 57.01 1022419684 0.00 0.00 > >> > Curl_str_key_compare > >> > 2.06 884.00 19.12 203749 0.00 0.00 create_conn > >> > 0.54 888.99 4.99 213306 0.00 0.00 ConnectionStore > >> > 0.49 893.50 4.51 7331251 0.00 0.00 dprintf_formatf > >> > 0.29 896.16 2.66 1287182329 0.00 0.00 > >> > ares__is_list_empty > >> > 0.23 898.28 2.12 locking_function > >> > 0.20 900.10 1.82 212210 0.00 0.00 > Curl_num_addresses > >> > 0.17 901.66 1.56 198385 0.00 0.00 > >> > curl_multi_remove_handle > >> > 0.12 902.81 1.15 411114 0.00 0.00 ares_cancel > >> > 0.12 903.95 1.14 3683448 0.00 0.00 multi_runsingle > >> > 0.11 904.97 1.03 13664120 0.00 0.00 > >> > client_tracing_function > >> > > >> > > >> > test 8: > >> > enable c-ares; > >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, 60); > >> > CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.24.0 > >> > > >> > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve > *.com > >> > to > >> > one IP address > >> > number of domain names in URLs: 780,000 > >> > number of client: 1000 > >> > CPU usage: 75% > >> > cpu% irq% sirq% sys% iowt% mem_used buf&cached > >> > 75.2 0.0 0.2 2.1 0.0 3537.2Mb 261.4Mb > >> > SSL TPS: 380 > >> > SSL throughput: 220Mbps > >> > > >> > # gprof /usr/bin/curl-loader gmon.out -p | head -20 > >> > Flat profile: > >> > Each sample counts as 0.01 seconds. > >> > % cumulative self self total > >> > time seconds seconds calls s/call s/call name > >> > 27.58 12.31 12.31 71140 0.00 0.00 > >> > Curl_hash_clean_with_criterium > >> > 19.09 20.83 8.52 176729692 0.00 0.00 > >> > hostcache_timestamp_remove > >> > 10.44 25.49 4.66 1589309 0.00 0.00 Curl_hash_pick > >> > 6.85 28.55 3.06 73204 0.00 0.00 create_conn > >> > 5.56 31.03 2.48 67620530 0.00 0.00 > >> > Curl_str_key_compare > >> > 4.41 33.00 1.97 200684 0.00 0.00 Curl_hash_add > >> > 3.56 34.59 1.59 2717956 0.00 0.00 dprintf_formatf > >> > 1.72 35.36 0.77 75247 0.00 0.00 ConnectionStore > >> > 1.70 36.12 0.76 444342256 0.00 0.00 > >> > ares__is_list_empty > >> > 1.23 36.67 0.55 locking_function > >> > 1.16 37.19 0.52 143342 0.00 0.00 ares_cancel > >> > 0.99 37.63 0.44 72211 0.00 0.00 > >> > curl_multi_remove_handle > >> > 0.74 37.96 0.33 1333371 0.00 0.00 multi_runsingle > >> > 0.68 38.27 0.31 101147798 0.00 0.00 addbyter > >> > 0.58 38.53 0.26 288737 0.00 0.00 > >> > curl_multi_socket_action > >> > > >> > > >> > > >> > test 9: > >> > enable c-ares; > >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, -1); > >> > CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.24.0 > >> > > >> > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve > *.com > >> > to > >> > one IP address > >> > number of domain names in URLs: 780,000 > >> > number of client: 1000 > >> > CPU usage: 75% > >> > cpu% irq% sirq% sys% iowt% mem_used buf&cached > >> > 75.1 0.0 0.6 4.1 0.0 3545.2Mb 260.1Mb > >> > SSL TPS: 380 > >> > SSL throughput: 200Mbps > >> > > >> > # gprof /usr/bin/curl-loader gmon.out -p | head -20 > >> > Flat profile: > >> > Each sample counts as 0.01 seconds. > >> > % cumulative self self total > >> > time seconds seconds calls s/call s/call name > >> > 34.00 31.13 31.13 72495 0.00 0.00 > >> > Curl_hash_clean_with_criterium > >> > 23.64 52.78 21.65 386070412 0.00 0.00 > >> > hostcache_timestamp_remove > >> > 13.28 64.94 12.16 1619440 0.00 0.00 Curl_hash_pick > >> > 5.92 70.36 5.42 137653129 0.00 0.00 > >> > Curl_str_key_compare > >> > 5.50 75.40 5.04 203018 0.00 0.00 Curl_hash_add > >> > 4.55 79.57 4.17 74618 0.00 0.00 create_conn > >> > 1.70 81.13 1.56 2757148 0.00 0.00 dprintf_formatf > >> > 0.98 82.03 0.90 76945 0.00 0.00 ConnectionStore > >> > 0.95 82.90 0.87 455775100 0.00 0.00 > >> > ares__is_list_empty > >> > 0.57 83.42 0.53 locking_function > >> > 0.39 83.78 0.36 146742 0.00 0.00 ares_cancel > >> > 0.35 84.10 0.32 102968806 0.00 0.00 addbyter > >> > 0.35 84.42 0.32 456587 0.00 0.00 Curl_readwrite > >> > 0.35 84.74 0.32 73133 0.00 0.00 > >> > curl_multi_remove_handle > >> > 0.34 85.05 0.31 1360545 0.00 0.00 multi_runsingle > >> > > >> > > >> > 2012/3/6 Robert Iakobashvili <cor...@gm...> > >> >> > >> >> If it will not be helpful this means that recent > >> >> changes in libcurl library have broken DNS interface. > >> >> > >> >> You can try a previos version and the latest libcurl, like: > >> >> > >> >> http://curl.haxx.se/download/curl-7.23.0.tar.gz > >> >> and > >> >> http://curl.haxx.se/downloadcurl-7.24.0.tar.gz > >> >> > >> >> by: > >> >> > >> >> 1. downloading a libcurl version and placing it to > >> >> curl-loader-0.56/packages ; > >> >> 2. correcting the Makefile in curl-loader-0.56 > >> >> CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.24 or CURL_VER:=7.23 > >> >> 3. make cleanall; make > >> >> > >> >> Please, send us your observations. > >> >> Thanks! > >> >> Robert > >> >> > >> >> On Mon, Mar 5, 2012 at 6:57 PM, Robert Iakobashvili > >> >> <cor...@gm...> > >> >> wrote: > >> >> > If so, please try to change > >> >> > > >> >> > loader.c, line 578 which looks like: > >> >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, -1); > >> >> > > >> >> > 1. instead of -1 try to set some value like 60 > >> >> > 2. make clean; make > >> >> > > >> >> > Thanks, > >> >> > Robert > >> >> > > >> >> > On Mon, Mar 5, 2012 at 6:19 PM, Fred Huang <di...@gm...> > wrote: > >> >> >> > >> >> >> Looks like the bottleneck comes from libcurl other than c-ares... > >> >> >> test 3 (no ares patch applied): > >> >> >> dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve > >> >> >> *.com > >> >> >> to one IP address > >> >> >> number of domain names in URLs: 780,000 > >> >> >> number of client: 1000 > >> >> >> CPU usage: 85% > >> >> >> cpu% irq% sirq% sys% iowt% mem_used buf&cached > >> >> >> 85.6 0.0 0.8 5.9 0.0 3334.3Mb 112.0Mb > >> >> >> SSL TPS: 320 > >> >> >> SSL throughput: 200Mbps > >> >> >> > >> >> >> # gprof /usr/bin/curl-loader gmon.out -p | head -50 > >> >> >> Flat profile: > >> >> >> Each sample counts as 0.01 seconds. > >> >> >> % cumulative self self total > >> >> >> time seconds seconds calls s/call s/call name > >> >> >> 40.80 120.16 120.16 116079 0.00 0.00 > >> >> >> Curl_hash_clean_with_criterium > >> >> >> 24.46 192.18 72.02 983166323 0.00 0.00 > >> >> >> hostcache_timestamp_remove > >> >> >> 13.30 231.35 39.17 2059341 0.00 0.00 > Curl_hash_pick > >> >> >> 6.45 250.35 19.00 240087 0.00 0.00 > Curl_hash_add > >> >> >> 4.81 264.51 14.16 334257102 0.00 0.00 > >> >> >> Curl_str_key_compare > >> >> >> 3.40 274.52 10.01 119514 0.00 0.00 create_conn > >> >> >> 0.93 277.25 2.73 4478910 0.00 0.00 > >> >> >> dprintf_formatf > >> >> >> 0.90 279.91 2.66 122209 0.00 0.00 > >> >> >> ConnectionStore > >> >> >> 0.40 281.09 1.18 > >> >> >> locking_function > >> >> >> 0.27 281.88 0.79 114759 0.00 0.00 > >> >> >> curl_multi_remove_handle > >> >> >> 0.21 282.51 0.63 7993664 0.00 0.00 > >> >> >> client_tracing_function > >> >> >> 0.21 283.12 0.61 122685 0.00 0.00 > >> >> >> Curl_num_addresses > >> >> >> 0.19 283.68 0.56 397028 0.00 0.00 > >> >> >> curl_multi_socket_action > >> >> >> 0.19 284.23 0.55 2036116 0.00 0.00 > >> >> >> multi_runsingle > >> >> >> 0.17 284.72 0.49 703687 0.00 0.00 > Curl_readwrite > >> >> >> 0.16 285.19 0.47 id_function > >> >> >> 0.15 285.62 0.43 29692029 0.00 0.00 > >> >> >> Curl_socket_check > >> >> >> 0.14 286.02 0.40 91973856 0.00 0.00 > >> >> >> Curl_raw_toupper > >> >> >> 0.13 286.40 0.38 164664144 0.00 0.00 addbyter > >> >> >> 0.13 286.77 0.37 2180797 0.00 0.00 Curl_infof > >> >> >> 0.11 287.10 0.33 122675 0.00 0.00 > >> >> >> Curl_if_is_interface_name > >> >> >> 0.10 287.39 0.29 471318 0.00 0.00 > >> >> >> ossl_connect_common > >> >> >> 0.10 287.67 0.28 2113528 0.00 0.00 > Curl_raw_equal > >> >> >> 0.09 287.93 0.26 8676 0.00 0.02 > >> >> >> curl_multi_socket_all > >> >> >> 0.08 288.17 0.24 1663262 0.00 0.00 Curl_splay > >> >> >> 0.08 288.41 0.24 8505 0.00 0.01 > >> >> >> curl_multi_perform > >> >> >> 0.07 288.63 0.22 3114576 0.00 0.00 > >> >> >> Curl_pgrsUpdate > >> >> >> 0.06 288.80 0.17 119637 0.00 0.00 > >> >> >> singleipconnect > >> >> >> 0.06 288.97 0.17 7994818 0.00 0.00 Curl_debug > >> >> >> 0.05 289.13 0.16 235820 0.00 0.00 event_del > >> >> >> 0.05 289.29 0.16 119393071 0.00 0.00 > curl_strequal > >> >> >> 0.05 289.45 0.16 18535473 0.00 0.00 > alloc_addbyter > >> >> >> 0.05 289.61 0.16 356074 0.00 0.00 > Curl_hash_str > >> >> >> 0.05 289.76 0.15 1797233 0.00 0.00 > fd_key_compare > >> >> >> 0.05 289.91 0.15 234966 0.00 0.00 > >> >> >> Curl_ssl_getsessionid > >> >> >> 0.04 290.04 0.13 585853 0.00 0.00 > >> >> >> Curl_llist_destroy > >> >> >> 0.04 290.16 0.12 240826 0.00 0.00 Curl_resolv > >> >> >> 0.03 290.26 0.10 7514328 0.00 0.00 > >> >> >> Curl_raw_nequal > >> >> >> 0.03 290.36 0.10 2214436 0.00 0.00 Curl_read > >> >> >> 0.03 290.46 0.10 116872 0.00 0.00 > >> >> >> Curl_http_readwrite_headers > >> >> >> 0.03 290.56 0.10 > ssl_tls_trace > >> >> >> 0.03 290.66 0.10 8013097 0.00 0.00 > scan_response > >> >> >> 0.03 290.75 0.09 29488555 0.00 0.00 curlx_tvdiff > >> >> >> 0.03 290.84 0.09 2142468 0.00 0.00 Curl_setopt > >> >> >> 0.03 290.93 0.09 476299 0.00 0.00 Curl_expire > >> >> >> > >> >> >> > >> >> >> > >> >> >> > >> >> >> > >> >> >> > >> >> >> > >> >> >> > >> >> >> 2012/3/5 Robert Iakobashvili <cor...@gm...> > >> >> >>> > >> >> >>> Dear Fred Huang, > >> >> >>> > >> >> >>> Thanks for your posting. > >> >> >>> > >> >> >>> Could you please check if this is due to some badness at cares? > >> >> >>> Thanks. > >> >> >>> > >> >> >>> Let's exclude usage of cares based DNS resolving by applying the > >> >> >>> patch > >> >> >>> attached > >> >> >>> and use instead global DNS cache. > >> >> >>> > >> >> >>> cd ./curl-loader-0.56 > >> >> >>> patch -p1 < path-to/disable-cares-use-global-dns.patch > >> >> >>> make cleanall; make > >> >> >>> > >> >> >>> See if it helps. > >> >> >>> > >> >> >>> Global cache, unfortunately, is not thread safe. > >> >> >>> > >> >> >>> If it helps, we'll try to restore the previous version of cares. > >> >> >>> > >> >> >>> Thanks, > >> >> >>> Robert > >> >> >>> > >> >> >>> On Mon, Mar 5, 2012 at 4:28 PM, Fred Huang <di...@gm...> > >> >> >>> wrote: > >> >> >>>> > >> >> >>>> test 1: > >> >> >>>> > >> >> >>>> dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, > resolve > >> >> >>>> *.com to one IP address > >> >> >>>> number of domain names in all https URLs: 780,000 > >> >> >>>> number of client: 1000 > >> >> >>>> CPU usage: 78% > >> >> >>>> cpu% irq% sirq% sys% iowt% mem_used buf&cached > >> >> >>>> 78.8 0.0 0.6 3.8 0.0 3640.9Mb 282.1Mb > >> >> >>>> SSL TPS: 300 > >> >> >>>> SSL throughput: 200Mbps > >> >> >>>> > >> >> >>>> # gprof /usr/bin/curl-loader gmon.out -p | head -50 > >> >> >>>> Flat profile: > >> >> >>>> Each sample counts as 0.01 seconds. > >> >> >>>> % cumulative self self total > >> >> >>>> time seconds seconds calls s/call s/call name > >> >> >>>> 38.55 127.10 127.10 130975 0.00 0.00 > >> >> >>>> Curl_hash_clean_with_criterium > >> >> >>>> 24.63 208.32 81.22 1247528407 0.00 0.00 > >> >> >>>> hostcache_timestamp_remove > >> >> >>>> 13.08 251.44 43.12 2791988 0.00 0.00 > >> >> >>>> Curl_hash_pick > >> >> >>>> 7.30 275.51 24.07 352490 0.00 0.00 > >> >> >>>> Curl_hash_add > >> >> >>>> 6.01 295.31 19.80 410760770 0.00 0.00 > >> >> >>>> Curl_str_key_compare > >> >> >>>> 2.95 305.04 9.73 131015 0.00 0.00 > create_conn > >> >> >>>> 0.88 307.94 2.90 4841390 0.00 0.00 > >> >> >>>> dprintf_formatf > >> >> >>>> 0.74 310.38 2.44 136036 0.00 0.00 > >> >> >>>> ConnectionStore > >> >> >>>> 0.48 311.96 1.58 814604911 0.00 0.00 > >> >> >>>> ares__is_list_empty > >> >> >>>> 0.36 313.16 1.20 > >> >> >>>> locking_function > >> >> >>>> 0.27 314.06 0.90 261608 0.00 0.00 > ares_cancel > >> >> >>>> 0.22 314.79 0.73 127417 0.00 0.00 > >> >> >>>> curl_multi_remove_handle > >> >> >>>> 0.19 315.40 0.62 8906301 0.00 0.00 > >> >> >>>> client_tracing_function > >> >> >>>> 0.17 315.97 0.57 135305 0.00 0.00 > >> >> >>>> Curl_num_addresses > >> >> >>>> 0.17 316.54 0.57 181937907 0.00 0.00 addbyter > >> >> >>>> 0.17 317.09 0.55 2374363 0.00 0.00 > >> >> >>>> multi_runsingle > >> >> >>>> 0.16 317.62 0.53 541954 0.00 0.00 > >> >> >>>> ares__init_list_node > >> >> >>>> 0.16 318.14 0.52 494412 0.00 0.00 > >> >> >>>> curl_multi_socket_action > >> >> >>>> 0.14 318.60 0.46 33578897 0.00 0.00 > >> >> >>>> Curl_socket_check > >> >> >>>> 0.13 319.04 0.44 102951052 0.00 0.00 > >> >> >>>> Curl_raw_toupper > >> >> >>>> 0.13 319.48 0.44 787558 0.00 0.00 > >> >> >>>> Curl_readwrite > >> >> >>>> 0.13 319.91 0.44 > id_function > >> >> >>>> 0.12 320.29 0.38 391390 0.00 0.00 > >> >> >>>> Curl_hash_str > >> >> >>>> 0.09 320.58 0.29 9216 0.00 0.00 > >> >> >>>> curl_multi_perform > >> >> >>>> 0.08 320.86 0.28 2350908 0.00 0.00 > >> >> >>>> Curl_raw_equal > >> >> >>>> 0.08 321.14 0.28 2019415 0.00 0.00 Curl_splay > >> >> >>>> 0.08 321.40 0.26 520625 0.00 0.00 > >> >> >>>> ossl_connect_common > >> >> >>>> 0.08 321.65 0.25 3532992 0.00 0.00 > >> >> >>>> Curl_pgrsUpdate > >> >> >>>> 0.07 321.87 0.22 131778537 0.00 0.00 > >> >> >>>> curl_strequal > >> >> >>>> 0.05 322.05 0.18 8954118 0.00 0.00 > >> >> >>>> scan_response > >> >> >>>> 0.05 322.23 0.18 533644 0.00 0.00 > >> >> >>>> ares_expand_name > >> >> >>>> 0.05 322.40 0.17 2477345 0.00 0.00 > >> >> >>>> fd_key_compare > >> >> >>>> 0.05 322.57 0.17 2429135 0.00 0.00 Curl_infof > >> >> >>>> 0.05 322.74 0.17 130966 0.00 0.00 > >> >> >>>> singleipconnect > >> >> >>>> 0.05 322.90 0.16 9292 0.00 0.00 > >> >> >>>> curl_multi_socket_all > >> >> >>>> 0.05 323.06 0.16 270137 0.00 0.00 > >> >> >>>> ares__get_hostent > >> >> >>>> 0.05 323.21 0.15 8943657 0.00 0.00 Curl_debug > >> >> >>>> 0.04 323.35 0.14 262493 0.00 0.00 > >> >> >>>> Curl_ssl_getsessionid > >> >> >>>> 0.04 323.48 0.13 566006 0.00 0.00 > >> >> >>>> socket_callback > >> >> >>>> 0.04 323.61 0.13 33983864 0.00 0.00 > curlx_tvdiff > >> >> >>>> 0.04 323.74 0.13 132853 0.00 0.00 > >> >> >>>> Curl_http_readwrite_headers > >> >> >>>> 0.04 323.86 0.12 344032 0.00 0.00 epoll_del > >> >> >>>> 0.04 323.98 0.12 223477 0.00 0.00 Curl_poll > >> >> >>>> 0.04 324.10 0.12 8526280 0.00 0.00 > >> >> >>>> Curl_raw_nequal > >> >> >>>> 0.03 324.21 0.11 348839 0.00 0.00 event_del > >> >> >>>> > >> >> >>>> > >> >> >>>> > >> >> >>>> > >> >> >>>> test 2: > >> >> >>>> > >> >> >>>> dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, > resolve > >> >> >>>> *.com to one IP address > >> >> >>>> number of domain names in all https URLs: 1 > >> >> >>>> number of client: 1000 > >> >> >>>> CPU usage: 75% > >> >> >>>> SSL TPS: 1300 > >> >> >>>> SSL throughput: 700Mbps > >> >> >>>> > >> >> >>>> # gprof /usr/bin/curl-loader gmon.out -p | head -50 > >> >> >>>> Flat profile: > >> >> >>>> Each sample counts as 0.01 seconds. > >> >> >>>> % cumulative self self total > >> >> >>>> time seconds seconds calls s/call s/call name > >> >> >>>> 8.98 2.45 2.45 5127312 0.00 0.00 > >> >> >>>> dprintf_formatf > >> >> >>>> 8.90 4.88 2.43 1292796503 0.00 0.00 > >> >> >>>> ares__is_list_empty > >> >> >>>> 6.12 6.55 1.67 322461 0.00 0.00 > create_conn > >> >> >>>> 4.58 7.80 1.25 419872 0.00 0.00 > ares_cancel > >> >> >>>> 4.10 8.92 1.12 2100162 0.00 0.00 > >> >> >>>> Curl_readwrite > >> >> >>>> 3.92 9.99 1.07 4527388 0.00 0.00 > >> >> >>>> Curl_hash_pick > >> >> >>>> 3.43 10.93 0.94 > >> >> >>>> locking_function > >> >> >>>> 3.04 11.76 0.83 909759 0.00 0.00 > >> >> >>>> curl_multi_socket_action > >> >> >>>> 3.00 12.58 0.82 97252 0.00 0.00 > >> >> >>>> Curl_hash_clean_with_criterium > >> >> >>>> 2.71 13.32 0.74 179704016 0.00 0.00 > >> >> >>>> Curl_raw_toupper > >> >> >>>> 2.68 14.05 0.73 4646524 0.00 0.00 > >> >> >>>> multi_runsingle > >> >> >>>> 2.44 14.71 0.67 13862992 0.00 0.00 > >> >> >>>> client_tracing_function > >> >> >>>> 2.42 15.37 0.66 319735 0.00 0.00 > >> >> >>>> curl_multi_remove_handle > >> >> >>>> 2.42 16.03 0.66 22 0.03 0.03 > >> >> >>>> ares__init_list_node > >> >> >>>> 1.94 16.56 0.53 9930244 0.00 0.00 > >> >> >>>> hostcache_timestamp_remove > >> >> >>>> 1.94 17.09 0.53 169320088 0.00 0.00 addbyter > >> >> >>>> 1.80 17.58 0.49 > id_function > >> >> >>>> 1.36 17.95 0.37 7057146 0.00 0.00 > >> >> >>>> Curl_pgrsUpdate > >> >> >>>> 1.36 18.32 0.37 16835 0.00 0.00 > >> >> >>>> curl_multi_socket_all > >> >> >>>> 1.25 18.66 0.34 16719 0.00 0.00 > >> >> >>>> curl_multi_perform > >> >> >>>> 1.21 18.99 0.33 329403 0.00 0.00 > >> >> >>>> Curl_http_readwrite_headers > >> >> >>>> 1.17 19.31 0.32 4036624 0.00 0.00 Curl_splay > >> >> >>>> 1.14 19.62 0.31 20838902 0.00 0.00 > >> >> >>>> Curl_raw_nequal > >> >> >>>> 0.82 19.85 0.23 2764123 0.00 0.00 Curl_infof > >> >> >>>> 0.81 20.07 0.22 5933857 0.00 0.00 ossl_recv > >> >> >>>> 0.77 20.28 0.21 323112 0.00 0.00 > >> >> >>>> Curl_splayremovebyaddr > >> >> >>>> 0.75 20.48 0.21 5943991 0.00 0.00 Curl_read > >> >> >>>> 0.70 20.67 0.19 425174 0.00 0.00 event_del > >> >> >>>> 0.70 20.86 0.19 98652 0.00 0.00 > >> >> >>>> Curl_if_is_interface_name > >> >> >>>> 0.70 21.05 0.19 8677733 0.00 0.00 > >> >> >>>> Curl_socket_check > >> >> >>>> 0.59 21.21 0.16 4129554 0.00 0.00 > >> >> >>>> fd_key_compare > >> >> >>>> 0.57 21.37 0.16 14017019 0.00 0.00 Curl_debug > >> >> >>>> 0.55 21.52 0.15 9376861 0.00 0.00 > >> >> >>>> stat_data_in_add > >> >> >>>> 0.55 21.67 0.15 6840468 0.00 0.00 > >> >> >>>> Curl_timeleft > >> >> >>>> 0.55 21.82 0.15 2054602 0.00 0.00 > >> >> >>>> Curl_raw_equal > >> >> >>>> 0.55 21.97 0.15 1291611 0.00 0.00 > Curl_expire > >> >> >>>> 0.51 22.11 0.14 16570063 0.00 0.00 > curlx_tvnow > >> >> >>>> 0.51 22.25 0.14 13863284 0.00 0.00 > >> >> >>>> scan_response > >> >> >>>> 0.51 22.39 0.14 5498864 0.00 0.00 > Curl_setopt > >> >> >>>> 0.48 22.52 0.13 97490291 0.00 0.00 > >> >> >>>> curl_strequal > >> >> >>>> 0.44 22.64 0.12 324253 0.00 0.00 Curl_http > >> >> >>>> 0.44 22.76 0.12 394386 0.00 0.00 > >> >> >>>> ossl_connect_common > >> >> >>>> 0.40 22.87 0.11 3581335 0.00 0.00 > Curl_getinfo > >> >> >>>> 0.38 22.97 0.11 26983092 0.00 0.00 > >> >> >>>> alloc_addbyter > >> >> >>>> 0.37 23.07 0.10 8729044 0.00 0.00 > >> >> >>>> Curl_client_write > >> >> >>>> > >> >> >>>> > >> >> >>>> > >> >> >>>> > >> >> >>>> > ------------------------------------------------------------------------------ > >> >> >>>> Try before you buy = See our experts in action! > >> >> >>>> The most comprehensive online learning library for Microsoft > >> >> >>>> developers > >> >> >>>> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, > CSS3, > >> >> >>>> MVC3, > >> >> >>>> Metro Style Apps, more. Free future releases when you subscribe > >> >> >>>> now! > >> >> >>>> http://p.sf.net/sfu/learndevnow-dev2 > >> >> >>>> _______________________________________________ > >> >> >>>> curl-loader-devel mailing list > >> >> >>>> cur...@li... > >> >> >>>> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel > >> >> >>>> > >> >> >>> > >> >> >>> > >> >> >>> > >> >> >>> -- > >> >> >>> Regards, > >> >> >>> Robert Iakobashvili, Ph.D. > >> >> >>> > >> >> >>> Home: http://www.ghotit.com > >> >> >>> Blog: http://www.ghotit.com/category/ghotit-blogs/ > >> >> >>> Twitter: http://twitter.com/ghotit > >> >> >>> Facebook: http://facebook.com/ghotit > >> >> >>> > >> >> >>> > ...................................................................... > >> >> >>> Ghotit Dyslexia > >> >> >>> Assistive technology that understands you > >> >> >>> > >> >> >>> > ...................................................................... > >> >> >>> > >> >> >>> > >> >> >>> > >> >> >>> > >> >> >>> > ------------------------------------------------------------------------------ > >> >> >>> Try before you buy = See our experts in action! > >> >> >>> The most comprehensive online learning library for Microsoft > >> >> >>> developers > >> >> >>> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, > CSS3, > >> >> >>> MVC3, > >> >> >>> Metro Style Apps, more. Free future releases when you subscribe > >> >> >>> now! > >> >> >>> http://p.sf.net/sfu/learndevnow-dev2 > >> >> >>> _______________________________________________ > >> >> >>> curl-loader-devel mailing list > >> >> >>> cur...@li... > >> >> >>> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel > >> >> >>> > >> >> >> > >> >> >> > >> >> >> > >> >> >> > >> >> >> > ------------------------------------------------------------------------------ > >> >> >> Try before you buy = See our experts in action! > >> >> >> The most comprehensive online learning library for Microsoft > >> >> >> developers > >> >> >> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, > >> >> >> MVC3, > >> >> >> Metro Style Apps, more. Free future releases when you subscribe > now! > >> >> >> http://p.sf.net/sfu/learndevnow-dev2 > >> >> >> _______________________________________________ > >> >> >> curl-loader-devel mailing list > >> >> >> cur...@li... > >> >> >> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel > >> >> >> > >> >> > > >> >> > > >> >> > > >> >> > -- > >> >> > Regards, > >> >> > Robert Iakobashvili, Ph.D. > >> >> > > >> >> > Home: http://www.ghotit.com > >> >> > Blog: http://www.ghotit.com/category/ghotit-blogs/ > >> >> > Twitter: http://twitter.com/ghotit > >> >> > Facebook: http://facebook.com/ghotit > >> >> > > >> >> > > ...................................................................... > >> >> > Ghotit Dyslexia > >> >> > Assistive technology that understands you > >> >> > > >> >> > > ...................................................................... > >> >> > >> >> > >> >> > >> >> -- > >> >> Regards, > >> >> Robert Iakobashvili, Ph.D. > >> >> > >> >> Home: http://www.ghotit.com > >> >> Blog: http://www.ghotit.com/category/ghotit-blogs/ > >> >> Twitter: http://twitter.com/ghotit > >> >> Facebook: http://facebook.com/ghotit > >> >> > ...................................................................... > >> >> Ghotit Dyslexia > >> >> Assistive technology that understands you > >> >> > ...................................................................... > >> >> > >> >> > >> >> > >> >> > ------------------------------------------------------------------------------ > >> >> Try before you buy = See our experts in action! > >> >> The most comprehensive online learning library for Microsoft > developers > >> >> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, > >> >> MVC3, > >> >> Metro Style Apps, more. Free future releases when you subscribe now! > >> >> http://p.sf.net/sfu/learndevnow-dev2 > >> >> _______________________________________________ > >> >> curl-loader-devel mailing list > >> >> cur...@li... > >> >> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel > >> > > >> > > >> > > >> > > >> > > ------------------------------------------------------------------------------ > >> > Keep Your Developer Skills Current with LearnDevNow! > >> > The most comprehensive online learning library for Microsoft > developers > >> > is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, > MVC3, > >> > Metro Style Apps, more. Free future releases when you subscribe now! > >> > http://p.sf.net/sfu/learndevnow-d2d > >> > _______________________________________________ > >> > curl-loader-devel mailing list > >> > cur...@li... > >> > https://lists.sourceforge.net/lists/listinfo/curl-loader-devel > >> > > >> > >> > >> > >> -- > >> Regards, > >> Robert Iakobashvili, Ph.D. > >> > >> Home: http://www.ghotit.com > >> Blog: http://www.ghotit.com/category/ghotit-blogs/ > >> Twitter: http://twitter.com/ghotit > >> Facebook: http://facebook.com/ghotit > >> ...................................................................... > >> Ghotit Dyslexia > >> Assistive technology that understands you > >> ...................................................................... > >> > >> > >> > ------------------------------------------------------------------------------ > >> Keep Your Developer Skills Current with LearnDevNow! > >> The most comprehensive online learning library for Microsoft developers > >> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, > >> Metro Style Apps, more. Free future releases when you subscribe now! > >> http://p.sf.net/sfu/learndevnow-d2d > >> _______________________________________________ > >> curl-loader-devel mailing list > >> cur...@li... > >> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel > > > > > > > > > ------------------------------------------------------------------------------ > > Keep Your Developer Skills Current with LearnDevNow! > > The most comprehensive online learning library for Microsoft developers > > is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, > > Metro Style Apps, more. Free future releases when you subscribe now! > > http://p.sf.net/sfu/learndevnow-d2d > > _______________________________________________ > > curl-loader-devel mailing list > > cur...@li... > > https://lists.sourceforge.net/lists/listinfo/curl-loader-devel > > > > > > -- > Regards, > Robert Iakobashvili, Ph.D. > > Home: http://www.ghotit.com > Blog: http://www.ghotit.com/category/ghotit-blogs/ > Twitter: http://twitter.com/ghotit > Facebook: http://facebook.com/ghotit > ...................................................................... > Ghotit Dyslexia > Assistive technology that understands you > ...................................................................... > > > ------------------------------------------------------------------------------ > Keep Your Developer Skills Current with LearnDevNow! > The most comprehensive online learning library for Microsoft developers > is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, > Metro Style Apps, more. Free future releases when you subscribe now! > http://p.sf.net/sfu/learndevnow-d2d > _______________________________________________ > curl-loader-devel mailing list > cur...@li... > https://lists.sourceforge.net/lists/listinfo/curl-loader-devel > |
|
From: Robert I. <cor...@gm...> - 2012-03-06 16:40:51
|
Dear Fred, If I understand correctly your observations, the issue is not in: 1. DNS resolving; 2. Version of cares or libcurl; but rather in Less effective handling of HTTPS versus HTTP in libcurl. Please, confirm that my understanding is correct. HTTPS URLs are using an extra locking in openssl with -t option that could be making the issue. If you are using -t option, try to load without it and let's say with 300 clients. If you are not using -t, thus the only way to proceed is to share your observations with libcurl people at libcurl development <cur...@co...> Thank you. Best wishes. Robert On Tue, Mar 6, 2012 at 6:17 PM, Fred Huang <di...@gm...> wrote: > 1. the major CPU consumer is curl-loader. here is the top: > top - 07:36:36 up 23:31, 5 users, load average: 4.32, 2.26, 0.91 > Tasks: 237 total, 2 running, 235 sleeping, 0 stopped, 0 zombie > Cpu0 : 76.6%us, 3.6%sy, 0.0%ni, 19.4%id, 0.0%wa, 0.0%hi, 0.3%si, 0.0% > Cpu1 : 72.4%us, 4.0%sy, 0.0%ni, 22.9%id, 0.0%wa, 0.0%hi, 0.7%si, 0.0% > Cpu2 : 65.8%us, 3.3%sy, 0.0%ni, 30.6%id, 0.0%wa, 0.0%hi, 0.3%si, 0.0% > Cpu3 : 75.2%us, 4.3%sy, 0.0%ni, 19.5%id, 0.0%wa, 0.0%hi, 1.0%si, 0.0% > Cpu4 : 79.1%us, 4.0%sy, 0.0%ni, 16.8%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0% > Cpu5 : 75.8%us, 4.3%sy, 0.0%ni, 19.2%id, 0.0%wa, 0.0%hi, 0.7%si, 0.0% > Cpu6 : 74.1%us, 5.0%sy, 0.0%ni, 20.3%id, 0.0%wa, 0.0%hi, 0.7%si, 0.0% > Cpu7 : 78.7%us, 4.0%sy, 0.0%ni, 16.6%id, 0.0%wa, 0.0%hi, 0.7%si, 0.0% > Mem: 16209932k total, 3829500k used, 12380432k free, 99112k buffers > Swap: 18448376k total, 0k used, 18448376k free, 320284k cached > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 30795 root 20 0 3200m 2.8g 2296 S 79.2 18.4 21:26.84 curl-loader > 1832 nobody 20 0 13944 1736 580 S 0.1 0.0 0:32.65 dnsmasq > 9 root 20 0 0 0 0 S 0.0 0.0 0:00.53 ksoftirqd/1 > 30825 root 20 0 15220 1360 960 R 0.0 0.0 0:00.42 top > 1 root 20 0 19400 1568 1256 S 0.0 0.0 0:00.75 init > 2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd > 3 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/0 > 4 root 20 0 0 0 0 S 0.0 0.0 0:00.06 ksoftirqd/0 > 5 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/0 > > > 2. 1000 loading clients > > 3. yes, tried 0.52 but no lucky -- CAPS 380, throughput 200Mbps, CPU 90% > top - 08:17:10 up 1 day, 12 min, 8 users, load average: 4.66, 3.39, 1.88 > Tasks: 245 total, 2 running, 243 sleeping, 0 stopped, 0 zombie > Cpu0 : 82.2%us, 4.0%sy, 0.0%ni, 13.9%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0% > Cpu1 : 86.1%us, 4.3%sy, 0.0%ni, 7.9%id, 0.0%wa, 0.0%hi, 1.7%si, 0.0% > Cpu2 : 85.3%us, 4.0%sy, 0.0%ni, 10.0%id, 0.0%wa, 0.0%hi, 0.7%si, 0.0% > Cpu3 : 81.8%us, 3.0%sy, 0.0%ni, 14.2%id, 0.0%wa, 0.0%hi, 1.0%si, 0.0% > Cpu4 : 83.7%us, 4.0%sy, 0.0%ni, 12.3%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0% > Cpu5 : 92.7%us, 3.3%sy, 0.0%ni, 3.3%id, 0.0%wa, 0.0%hi, 0.7%si, 0.0% > Cpu6 : 89.7%us, 2.0%sy, 0.0%ni, 8.0%id, 0.0%wa, 0.0%hi, 0.3%si, 0.0% > Cpu7 : 88.1%us, 3.6%sy, 0.0%ni, 6.6%id, 0.0%wa, 0.0%hi, 1.7%si, 0.0% > Mem: 16209932k total, 1795748k used, 14414184k free, 102824k buffers > Swap: 18448376k total, 0k used, 18448376k free, 382988k cached > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 12236 root 20 0 1150m 894m 2572 S 90.7 5.7 51:11.91 curl-loader > 1832 nobody 20 0 13944 1736 580 S 0.1 0.0 0:47.04 dnsmasq > > ]# gprof /usr/bin/curl-loader gmon.out -p | head -20 > Flat profile: > Each sample counts as 0.01 seconds. > % cumulative self self total > time seconds seconds calls s/call s/call name > 43.92 151.95 151.95 157295 0.00 0.00 > Curl_hash_clean_with_criterium > 22.90 231.17 79.22 1712190971 0.00 0.00 > hostcache_timestamp_remove > 13.15 276.68 45.51 3260321 0.00 0.00 Curl_hash_pick > 7.51 302.65 25.97 394171 0.00 0.00 Curl_hash_add > 5.98 323.34 20.68 567437518 0.00 0.00 Curl_str_key_compare > 1.00 326.80 3.46 5380220 0.00 0.00 dprintf_formatf > 0.41 328.21 1.41 locking_function > 0.32 329.32 1.11 469222138 0.00 0.00 ares__is_list_empty > 0.31 330.40 1.08 3284508 0.00 0.00 multi_runsingle > 0.28 331.38 0.98 10006806 0.00 0.00 > client_tracing_function > 0.24 332.21 0.83 151951 0.00 0.00 Curl_num_addresses > 0.22 332.97 0.77 188075672 0.00 0.00 addbyter > 0.19 333.64 0.67 841840 0.00 0.00 Curl_readwrite > 0.19 334.29 0.65 id_function > 0.18 334.90 0.61 39843 0.00 0.01 curl_multi_socket_all > > > > 4. yes, it is correct for 0.56. If use same URL list by replacing https to > http, I can get CAPS 1500 and throughput 870Mbps with CPU% < 60% > > > 2012/3/6 Robert Iakobashvili <cor...@gm...> >> >> Several questions: >> >> 1. Could you please run top and see if there are other processes burning >> CPU? >> 2. What is the number of loading clients that you use? >> 3. Could you try curl-loader-0.52 if it compiles for you? >> If not, please try to place curl-loader-7.19.7.tar.gz and >> c-ares-16.0.tar.gz, >> update their versions in Makefile and rebuild by make cleanall; make >> >> 4. Is it correct to say that for HTTP URLs (not HTTPS) you are getting >> a good throughput >> and CAPS with curl-loader-0.56? >> >> Thanks, >> Robert >> >> On Tue, Mar 6, 2012 at 3:52 AM, Fred Huang <di...@gm...> wrote: >> > TIMEOUT -1 -> 60 works a little better but no performance improvement on >> > CAPS or throughput. >> > >> > test 5: >> > enable c-ares; >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, 60); >> > >> > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve *.com >> > to >> > one IP address >> > number of domain names in URLs: 780,000 >> > number of client: 1000 >> > CPU usage: 74% >> > cpu% irq% sirq% sys% iowt% mem_used buf&cached >> > 74.0 0.0 0.7 4.2 0.0 3530.6Mb 250.8Mb >> > SSL TPS: 360 >> > SSL throughput: 220Mbps >> > >> > # gprof /usr/bin/curl-loader gmon.out -p | head -50 >> > Flat profile: >> > Each sample counts as 0.01 seconds. >> > % cumulative self self total >> > time seconds seconds calls s/call s/call name >> > 26.63 23.13 23.13 119226 0.00 0.00 >> > Curl_hash_clean_with_criterium >> > 20.34 40.80 17.67 317792146 0.00 0.00 >> > hostcache_timestamp_remove >> > 11.27 50.59 9.79 2579903 0.00 0.00 Curl_hash_pick >> > 7.84 57.40 6.81 120606 0.00 0.00 create_conn >> > 5.65 62.31 4.91 119117401 0.00 0.00 >> > Curl_str_key_compare >> > 4.98 66.64 4.33 329829 0.00 0.00 Curl_hash_add >> > 2.80 69.08 2.44 4476606 0.00 0.00 dprintf_formatf >> > 2.57 71.31 2.23 124019 0.00 0.00 ConnectionStore >> > 1.50 72.61 1.30 740997609 0.00 0.00 >> > ares__is_list_empty >> > 1.16 73.61 1.01 locking_function >> > 0.76 74.27 0.66 238744 0.00 0.00 ares_cancel >> > 0.70 74.88 0.61 457564 0.00 0.00 >> > curl_multi_socket_action >> > 0.67 75.46 0.58 118616 0.00 0.00 >> > curl_multi_remove_handle >> > 0.58 75.97 0.51 8211502 0.00 0.00 >> > client_tracing_function >> > 0.53 76.43 0.46 733561 0.00 0.00 Curl_readwrite >> > 0.51 76.87 0.44 2183179 0.00 0.00 multi_runsingle >> > 0.51 77.31 0.44 166585567 0.00 0.00 addbyter >> > 0.44 77.69 0.38 2175805 0.00 0.00 Curl_raw_equal >> > 0.43 78.06 0.38 id_function >> > 0.40 78.41 0.35 29789346 0.00 0.00 Curl_socket_check >> > 0.40 78.76 0.35 479671 0.00 0.00 >> > ossl_connect_common >> > 0.40 79.11 0.35 490639 0.00 0.00 >> > ares__init_list_node >> > 0.33 79.40 0.29 94197670 0.00 0.00 Curl_raw_toupper >> > 0.33 79.69 0.29 3268031 0.00 0.00 Curl_pgrsUpdate >> > 0.28 79.93 0.24 8797 0.00 0.00 curl_multi_perform >> > 0.27 80.17 0.24 2233641 0.00 0.00 Curl_infof >> > 0.26 80.40 0.23 1839837 0.00 0.00 Curl_splay >> > 0.24 80.61 0.21 121713028 0.00 0.00 curl_strequal >> > 0.21 80.79 0.18 243333 0.00 0.00 >> > Curl_ssl_getsessionid >> > 0.18 80.95 0.16 122180 0.00 0.00 Curl_num_addresses >> > 0.17 81.09 0.15 8271666 0.00 0.00 Curl_debug >> > 0.16 81.23 0.14 18902452 0.00 0.00 alloc_addbyter >> > 0.16 81.37 0.14 8851 0.00 0.00 >> > curl_multi_socket_all >> > 0.14 81.50 0.13 2236769 0.00 0.00 Curl_read >> > 0.14 81.62 0.12 2271700 0.00 0.00 fd_key_compare >> > 0.14 81.74 0.12 323167 0.00 0.00 event_del >> > 0.14 81.86 0.12 205436 0.00 0.00 read_udp_packets >> > 0.14 81.98 0.12 200979 0.00 0.00 processfds >> > 0.13 82.09 0.11 123886 0.00 0.00 >> > Curl_if_is_interface_name >> > 0.13 82.20 0.11 120350 0.00 0.00 Curl_http >> > 0.13 82.31 0.11 ssl_tls_trace >> > 0.12 82.41 0.10 119229 0.00 0.00 >> > Curl_splayremovebyaddr >> > 0.12 82.51 0.10 7798717 0.00 0.00 Curl_raw_nequal >> > 0.12 82.61 0.10 784485 0.00 0.00 event_add >> > 0.10 82.70 0.09 643482 0.00 0.00 Curl_expire >> > >> > >> > >> > >> > >> > test 6: >> > enable c-ares; >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, 60); >> > CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.23.0 >> > >> > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve *.com >> > to >> > one IP address >> > number of domain names in URLs: 780,000 >> > number of client: 1000 >> > CPU usage: 86% >> > cpu% irq% sirq% sys% iowt% mem_used buf&cached >> > 85.9 0.0 0.2 2.1 0.0 3537.2Mb 261.4Mb >> > SSL TPS: 360 >> > SSL throughput: 200Mbps >> > >> > # gprof /usr/bin/curl-loader gmon.out -p | head -20 >> > Flat profile: >> > Each sample counts as 0.01 seconds. >> > % cumulative self self total >> > time seconds seconds calls s/call s/call name >> > 27.19 19.96 19.96 103472 0.00 0.00 >> > Curl_hash_clean_with_criterium >> > 21.73 35.91 15.95 274168981 0.00 0.00 >> > hostcache_timestamp_remove >> > 10.33 43.49 7.58 2256441 0.00 0.00 Curl_hash_pick >> > 7.21 48.78 5.29 104540 0.00 0.00 create_conn >> > 6.20 53.33 4.55 103413405 0.00 0.00 >> > Curl_str_key_compare >> > 4.88 56.91 3.58 287109 0.00 0.00 Curl_hash_add >> > 3.02 59.13 2.22 3797855 0.00 0.00 dprintf_formatf >> > 2.10 60.67 1.54 107711 0.00 0.00 ConnectionStore >> > 1.65 61.88 1.21 642725327 0.00 0.00 >> > ares__is_list_empty >> > 1.19 62.76 0.88 locking_function >> > 0.87 63.40 0.64 207817 0.00 0.00 ares_cancel >> > 0.56 63.81 0.41 404108 0.00 0.00 >> > curl_multi_socket_action >> > 0.54 64.21 0.40 425603 0.00 0.00 >> > ares__init_list_node >> > 0.53 64.60 0.39 7068112 0.00 0.00 >> > client_tracing_function >> > 0.49 64.96 0.36 1908587 0.00 0.00 multi_runsingle >> > >> > >> > >> > test 7: >> > enable c-ares; >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, -1); >> > CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.23.0 >> > >> > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve *.com >> > to >> > one IP address >> > number of domain names in URLs: 780,000 >> > number of client: 1000 >> > CPU usage: 86% >> > cpu% irq% sirq% sys% iowt% mem_used buf&cached >> > 86.6 0.0 0.6 3.3 0.0 3624.8Mb 263.0Mb >> > SSL TPS: 300 >> > SSL throughput: 200Mbps >> > >> > # gprof /usr/bin/curl-loader gmon.out -p | head -20 >> > Flat profile: >> > Each sample counts as 0.01 seconds. >> > % cumulative self self total >> > time seconds seconds calls s/call s/call name >> > 39.83 369.71 369.71 207173 0.00 0.00 >> > Curl_hash_clean_with_criterium >> > 24.27 594.98 225.27 3198337733 0.00 0.00 >> > hostcache_timestamp_remove >> > 13.75 722.59 127.61 4312864 0.00 0.00 Curl_hash_pick >> > 9.19 807.87 85.28 546507 0.00 0.00 Curl_hash_add >> > 6.14 864.88 57.01 1022419684 0.00 0.00 >> > Curl_str_key_compare >> > 2.06 884.00 19.12 203749 0.00 0.00 create_conn >> > 0.54 888.99 4.99 213306 0.00 0.00 ConnectionStore >> > 0.49 893.50 4.51 7331251 0.00 0.00 dprintf_formatf >> > 0.29 896.16 2.66 1287182329 0.00 0.00 >> > ares__is_list_empty >> > 0.23 898.28 2.12 locking_function >> > 0.20 900.10 1.82 212210 0.00 0.00 Curl_num_addresses >> > 0.17 901.66 1.56 198385 0.00 0.00 >> > curl_multi_remove_handle >> > 0.12 902.81 1.15 411114 0.00 0.00 ares_cancel >> > 0.12 903.95 1.14 3683448 0.00 0.00 multi_runsingle >> > 0.11 904.97 1.03 13664120 0.00 0.00 >> > client_tracing_function >> > >> > >> > test 8: >> > enable c-ares; >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, 60); >> > CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.24.0 >> > >> > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve *.com >> > to >> > one IP address >> > number of domain names in URLs: 780,000 >> > number of client: 1000 >> > CPU usage: 75% >> > cpu% irq% sirq% sys% iowt% mem_used buf&cached >> > 75.2 0.0 0.2 2.1 0.0 3537.2Mb 261.4Mb >> > SSL TPS: 380 >> > SSL throughput: 220Mbps >> > >> > # gprof /usr/bin/curl-loader gmon.out -p | head -20 >> > Flat profile: >> > Each sample counts as 0.01 seconds. >> > % cumulative self self total >> > time seconds seconds calls s/call s/call name >> > 27.58 12.31 12.31 71140 0.00 0.00 >> > Curl_hash_clean_with_criterium >> > 19.09 20.83 8.52 176729692 0.00 0.00 >> > hostcache_timestamp_remove >> > 10.44 25.49 4.66 1589309 0.00 0.00 Curl_hash_pick >> > 6.85 28.55 3.06 73204 0.00 0.00 create_conn >> > 5.56 31.03 2.48 67620530 0.00 0.00 >> > Curl_str_key_compare >> > 4.41 33.00 1.97 200684 0.00 0.00 Curl_hash_add >> > 3.56 34.59 1.59 2717956 0.00 0.00 dprintf_formatf >> > 1.72 35.36 0.77 75247 0.00 0.00 ConnectionStore >> > 1.70 36.12 0.76 444342256 0.00 0.00 >> > ares__is_list_empty >> > 1.23 36.67 0.55 locking_function >> > 1.16 37.19 0.52 143342 0.00 0.00 ares_cancel >> > 0.99 37.63 0.44 72211 0.00 0.00 >> > curl_multi_remove_handle >> > 0.74 37.96 0.33 1333371 0.00 0.00 multi_runsingle >> > 0.68 38.27 0.31 101147798 0.00 0.00 addbyter >> > 0.58 38.53 0.26 288737 0.00 0.00 >> > curl_multi_socket_action >> > >> > >> > >> > test 9: >> > enable c-ares; >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, -1); >> > CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.24.0 >> > >> > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve *.com >> > to >> > one IP address >> > number of domain names in URLs: 780,000 >> > number of client: 1000 >> > CPU usage: 75% >> > cpu% irq% sirq% sys% iowt% mem_used buf&cached >> > 75.1 0.0 0.6 4.1 0.0 3545.2Mb 260.1Mb >> > SSL TPS: 380 >> > SSL throughput: 200Mbps >> > >> > # gprof /usr/bin/curl-loader gmon.out -p | head -20 >> > Flat profile: >> > Each sample counts as 0.01 seconds. >> > % cumulative self self total >> > time seconds seconds calls s/call s/call name >> > 34.00 31.13 31.13 72495 0.00 0.00 >> > Curl_hash_clean_with_criterium >> > 23.64 52.78 21.65 386070412 0.00 0.00 >> > hostcache_timestamp_remove >> > 13.28 64.94 12.16 1619440 0.00 0.00 Curl_hash_pick >> > 5.92 70.36 5.42 137653129 0.00 0.00 >> > Curl_str_key_compare >> > 5.50 75.40 5.04 203018 0.00 0.00 Curl_hash_add >> > 4.55 79.57 4.17 74618 0.00 0.00 create_conn >> > 1.70 81.13 1.56 2757148 0.00 0.00 dprintf_formatf >> > 0.98 82.03 0.90 76945 0.00 0.00 ConnectionStore >> > 0.95 82.90 0.87 455775100 0.00 0.00 >> > ares__is_list_empty >> > 0.57 83.42 0.53 locking_function >> > 0.39 83.78 0.36 146742 0.00 0.00 ares_cancel >> > 0.35 84.10 0.32 102968806 0.00 0.00 addbyter >> > 0.35 84.42 0.32 456587 0.00 0.00 Curl_readwrite >> > 0.35 84.74 0.32 73133 0.00 0.00 >> > curl_multi_remove_handle >> > 0.34 85.05 0.31 1360545 0.00 0.00 multi_runsingle >> > >> > >> > 2012/3/6 Robert Iakobashvili <cor...@gm...> >> >> >> >> If it will not be helpful this means that recent >> >> changes in libcurl library have broken DNS interface. >> >> >> >> You can try a previos version and the latest libcurl, like: >> >> >> >> http://curl.haxx.se/download/curl-7.23.0.tar.gz >> >> and >> >> http://curl.haxx.se/downloadcurl-7.24.0.tar.gz >> >> >> >> by: >> >> >> >> 1. downloading a libcurl version and placing it to >> >> curl-loader-0.56/packages ; >> >> 2. correcting the Makefile in curl-loader-0.56 >> >> CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.24 or CURL_VER:=7.23 >> >> 3. make cleanall; make >> >> >> >> Please, send us your observations. >> >> Thanks! >> >> Robert >> >> >> >> On Mon, Mar 5, 2012 at 6:57 PM, Robert Iakobashvili >> >> <cor...@gm...> >> >> wrote: >> >> > If so, please try to change >> >> > >> >> > loader.c, line 578 which looks like: >> >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, -1); >> >> > >> >> > 1. instead of -1 try to set some value like 60 >> >> > 2. make clean; make >> >> > >> >> > Thanks, >> >> > Robert >> >> > >> >> > On Mon, Mar 5, 2012 at 6:19 PM, Fred Huang <di...@gm...> wrote: >> >> >> >> >> >> Looks like the bottleneck comes from libcurl other than c-ares... >> >> >> test 3 (no ares patch applied): >> >> >> dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve >> >> >> *.com >> >> >> to one IP address >> >> >> number of domain names in URLs: 780,000 >> >> >> number of client: 1000 >> >> >> CPU usage: 85% >> >> >> cpu% irq% sirq% sys% iowt% mem_used buf&cached >> >> >> 85.6 0.0 0.8 5.9 0.0 3334.3Mb 112.0Mb >> >> >> SSL TPS: 320 >> >> >> SSL throughput: 200Mbps >> >> >> >> >> >> # gprof /usr/bin/curl-loader gmon.out -p | head -50 >> >> >> Flat profile: >> >> >> Each sample counts as 0.01 seconds. >> >> >> % cumulative self self total >> >> >> time seconds seconds calls s/call s/call name >> >> >> 40.80 120.16 120.16 116079 0.00 0.00 >> >> >> Curl_hash_clean_with_criterium >> >> >> 24.46 192.18 72.02 983166323 0.00 0.00 >> >> >> hostcache_timestamp_remove >> >> >> 13.30 231.35 39.17 2059341 0.00 0.00 Curl_hash_pick >> >> >> 6.45 250.35 19.00 240087 0.00 0.00 Curl_hash_add >> >> >> 4.81 264.51 14.16 334257102 0.00 0.00 >> >> >> Curl_str_key_compare >> >> >> 3.40 274.52 10.01 119514 0.00 0.00 create_conn >> >> >> 0.93 277.25 2.73 4478910 0.00 0.00 >> >> >> dprintf_formatf >> >> >> 0.90 279.91 2.66 122209 0.00 0.00 >> >> >> ConnectionStore >> >> >> 0.40 281.09 1.18 >> >> >> locking_function >> >> >> 0.27 281.88 0.79 114759 0.00 0.00 >> >> >> curl_multi_remove_handle >> >> >> 0.21 282.51 0.63 7993664 0.00 0.00 >> >> >> client_tracing_function >> >> >> 0.21 283.12 0.61 122685 0.00 0.00 >> >> >> Curl_num_addresses >> >> >> 0.19 283.68 0.56 397028 0.00 0.00 >> >> >> curl_multi_socket_action >> >> >> 0.19 284.23 0.55 2036116 0.00 0.00 >> >> >> multi_runsingle >> >> >> 0.17 284.72 0.49 703687 0.00 0.00 Curl_readwrite >> >> >> 0.16 285.19 0.47 id_function >> >> >> 0.15 285.62 0.43 29692029 0.00 0.00 >> >> >> Curl_socket_check >> >> >> 0.14 286.02 0.40 91973856 0.00 0.00 >> >> >> Curl_raw_toupper >> >> >> 0.13 286.40 0.38 164664144 0.00 0.00 addbyter >> >> >> 0.13 286.77 0.37 2180797 0.00 0.00 Curl_infof >> >> >> 0.11 287.10 0.33 122675 0.00 0.00 >> >> >> Curl_if_is_interface_name >> >> >> 0.10 287.39 0.29 471318 0.00 0.00 >> >> >> ossl_connect_common >> >> >> 0.10 287.67 0.28 2113528 0.00 0.00 Curl_raw_equal >> >> >> 0.09 287.93 0.26 8676 0.00 0.02 >> >> >> curl_multi_socket_all >> >> >> 0.08 288.17 0.24 1663262 0.00 0.00 Curl_splay >> >> >> 0.08 288.41 0.24 8505 0.00 0.01 >> >> >> curl_multi_perform >> >> >> 0.07 288.63 0.22 3114576 0.00 0.00 >> >> >> Curl_pgrsUpdate >> >> >> 0.06 288.80 0.17 119637 0.00 0.00 >> >> >> singleipconnect >> >> >> 0.06 288.97 0.17 7994818 0.00 0.00 Curl_debug >> >> >> 0.05 289.13 0.16 235820 0.00 0.00 event_del >> >> >> 0.05 289.29 0.16 119393071 0.00 0.00 curl_strequal >> >> >> 0.05 289.45 0.16 18535473 0.00 0.00 alloc_addbyter >> >> >> 0.05 289.61 0.16 356074 0.00 0.00 Curl_hash_str >> >> >> 0.05 289.76 0.15 1797233 0.00 0.00 fd_key_compare >> >> >> 0.05 289.91 0.15 234966 0.00 0.00 >> >> >> Curl_ssl_getsessionid >> >> >> 0.04 290.04 0.13 585853 0.00 0.00 >> >> >> Curl_llist_destroy >> >> >> 0.04 290.16 0.12 240826 0.00 0.00 Curl_resolv >> >> >> 0.03 290.26 0.10 7514328 0.00 0.00 >> >> >> Curl_raw_nequal >> >> >> 0.03 290.36 0.10 2214436 0.00 0.00 Curl_read >> >> >> 0.03 290.46 0.10 116872 0.00 0.00 >> >> >> Curl_http_readwrite_headers >> >> >> 0.03 290.56 0.10 ssl_tls_trace >> >> >> 0.03 290.66 0.10 8013097 0.00 0.00 scan_response >> >> >> 0.03 290.75 0.09 29488555 0.00 0.00 curlx_tvdiff >> >> >> 0.03 290.84 0.09 2142468 0.00 0.00 Curl_setopt >> >> >> 0.03 290.93 0.09 476299 0.00 0.00 Curl_expire >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> 2012/3/5 Robert Iakobashvili <cor...@gm...> >> >> >>> >> >> >>> Dear Fred Huang, >> >> >>> >> >> >>> Thanks for your posting. >> >> >>> >> >> >>> Could you please check if this is due to some badness at cares? >> >> >>> Thanks. >> >> >>> >> >> >>> Let's exclude usage of cares based DNS resolving by applying the >> >> >>> patch >> >> >>> attached >> >> >>> and use instead global DNS cache. >> >> >>> >> >> >>> cd ./curl-loader-0.56 >> >> >>> patch -p1 < path-to/disable-cares-use-global-dns.patch >> >> >>> make cleanall; make >> >> >>> >> >> >>> See if it helps. >> >> >>> >> >> >>> Global cache, unfortunately, is not thread safe. >> >> >>> >> >> >>> If it helps, we'll try to restore the previous version of cares. >> >> >>> >> >> >>> Thanks, >> >> >>> Robert >> >> >>> >> >> >>> On Mon, Mar 5, 2012 at 4:28 PM, Fred Huang <di...@gm...> >> >> >>> wrote: >> >> >>>> >> >> >>>> test 1: >> >> >>>> >> >> >>>> dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve >> >> >>>> *.com to one IP address >> >> >>>> number of domain names in all https URLs: 780,000 >> >> >>>> number of client: 1000 >> >> >>>> CPU usage: 78% >> >> >>>> cpu% irq% sirq% sys% iowt% mem_used buf&cached >> >> >>>> 78.8 0.0 0.6 3.8 0.0 3640.9Mb 282.1Mb >> >> >>>> SSL TPS: 300 >> >> >>>> SSL throughput: 200Mbps >> >> >>>> >> >> >>>> # gprof /usr/bin/curl-loader gmon.out -p | head -50 >> >> >>>> Flat profile: >> >> >>>> Each sample counts as 0.01 seconds. >> >> >>>> % cumulative self self total >> >> >>>> time seconds seconds calls s/call s/call name >> >> >>>> 38.55 127.10 127.10 130975 0.00 0.00 >> >> >>>> Curl_hash_clean_with_criterium >> >> >>>> 24.63 208.32 81.22 1247528407 0.00 0.00 >> >> >>>> hostcache_timestamp_remove >> >> >>>> 13.08 251.44 43.12 2791988 0.00 0.00 >> >> >>>> Curl_hash_pick >> >> >>>> 7.30 275.51 24.07 352490 0.00 0.00 >> >> >>>> Curl_hash_add >> >> >>>> 6.01 295.31 19.80 410760770 0.00 0.00 >> >> >>>> Curl_str_key_compare >> >> >>>> 2.95 305.04 9.73 131015 0.00 0.00 create_conn >> >> >>>> 0.88 307.94 2.90 4841390 0.00 0.00 >> >> >>>> dprintf_formatf >> >> >>>> 0.74 310.38 2.44 136036 0.00 0.00 >> >> >>>> ConnectionStore >> >> >>>> 0.48 311.96 1.58 814604911 0.00 0.00 >> >> >>>> ares__is_list_empty >> >> >>>> 0.36 313.16 1.20 >> >> >>>> locking_function >> >> >>>> 0.27 314.06 0.90 261608 0.00 0.00 ares_cancel >> >> >>>> 0.22 314.79 0.73 127417 0.00 0.00 >> >> >>>> curl_multi_remove_handle >> >> >>>> 0.19 315.40 0.62 8906301 0.00 0.00 >> >> >>>> client_tracing_function >> >> >>>> 0.17 315.97 0.57 135305 0.00 0.00 >> >> >>>> Curl_num_addresses >> >> >>>> 0.17 316.54 0.57 181937907 0.00 0.00 addbyter >> >> >>>> 0.17 317.09 0.55 2374363 0.00 0.00 >> >> >>>> multi_runsingle >> >> >>>> 0.16 317.62 0.53 541954 0.00 0.00 >> >> >>>> ares__init_list_node >> >> >>>> 0.16 318.14 0.52 494412 0.00 0.00 >> >> >>>> curl_multi_socket_action >> >> >>>> 0.14 318.60 0.46 33578897 0.00 0.00 >> >> >>>> Curl_socket_check >> >> >>>> 0.13 319.04 0.44 102951052 0.00 0.00 >> >> >>>> Curl_raw_toupper >> >> >>>> 0.13 319.48 0.44 787558 0.00 0.00 >> >> >>>> Curl_readwrite >> >> >>>> 0.13 319.91 0.44 id_function >> >> >>>> 0.12 320.29 0.38 391390 0.00 0.00 >> >> >>>> Curl_hash_str >> >> >>>> 0.09 320.58 0.29 9216 0.00 0.00 >> >> >>>> curl_multi_perform >> >> >>>> 0.08 320.86 0.28 2350908 0.00 0.00 >> >> >>>> Curl_raw_equal >> >> >>>> 0.08 321.14 0.28 2019415 0.00 0.00 Curl_splay >> >> >>>> 0.08 321.40 0.26 520625 0.00 0.00 >> >> >>>> ossl_connect_common >> >> >>>> 0.08 321.65 0.25 3532992 0.00 0.00 >> >> >>>> Curl_pgrsUpdate >> >> >>>> 0.07 321.87 0.22 131778537 0.00 0.00 >> >> >>>> curl_strequal >> >> >>>> 0.05 322.05 0.18 8954118 0.00 0.00 >> >> >>>> scan_response >> >> >>>> 0.05 322.23 0.18 533644 0.00 0.00 >> >> >>>> ares_expand_name >> >> >>>> 0.05 322.40 0.17 2477345 0.00 0.00 >> >> >>>> fd_key_compare >> >> >>>> 0.05 322.57 0.17 2429135 0.00 0.00 Curl_infof >> >> >>>> 0.05 322.74 0.17 130966 0.00 0.00 >> >> >>>> singleipconnect >> >> >>>> 0.05 322.90 0.16 9292 0.00 0.00 >> >> >>>> curl_multi_socket_all >> >> >>>> 0.05 323.06 0.16 270137 0.00 0.00 >> >> >>>> ares__get_hostent >> >> >>>> 0.05 323.21 0.15 8943657 0.00 0.00 Curl_debug >> >> >>>> 0.04 323.35 0.14 262493 0.00 0.00 >> >> >>>> Curl_ssl_getsessionid >> >> >>>> 0.04 323.48 0.13 566006 0.00 0.00 >> >> >>>> socket_callback >> >> >>>> 0.04 323.61 0.13 33983864 0.00 0.00 curlx_tvdiff >> >> >>>> 0.04 323.74 0.13 132853 0.00 0.00 >> >> >>>> Curl_http_readwrite_headers >> >> >>>> 0.04 323.86 0.12 344032 0.00 0.00 epoll_del >> >> >>>> 0.04 323.98 0.12 223477 0.00 0.00 Curl_poll >> >> >>>> 0.04 324.10 0.12 8526280 0.00 0.00 >> >> >>>> Curl_raw_nequal >> >> >>>> 0.03 324.21 0.11 348839 0.00 0.00 event_del >> >> >>>> >> >> >>>> >> >> >>>> >> >> >>>> >> >> >>>> test 2: >> >> >>>> >> >> >>>> dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve >> >> >>>> *.com to one IP address >> >> >>>> number of domain names in all https URLs: 1 >> >> >>>> number of client: 1000 >> >> >>>> CPU usage: 75% >> >> >>>> SSL TPS: 1300 >> >> >>>> SSL throughput: 700Mbps >> >> >>>> >> >> >>>> # gprof /usr/bin/curl-loader gmon.out -p | head -50 >> >> >>>> Flat profile: >> >> >>>> Each sample counts as 0.01 seconds. >> >> >>>> % cumulative self self total >> >> >>>> time seconds seconds calls s/call s/call name >> >> >>>> 8.98 2.45 2.45 5127312 0.00 0.00 >> >> >>>> dprintf_formatf >> >> >>>> 8.90 4.88 2.43 1292796503 0.00 0.00 >> >> >>>> ares__is_list_empty >> >> >>>> 6.12 6.55 1.67 322461 0.00 0.00 create_conn >> >> >>>> 4.58 7.80 1.25 419872 0.00 0.00 ares_cancel >> >> >>>> 4.10 8.92 1.12 2100162 0.00 0.00 >> >> >>>> Curl_readwrite >> >> >>>> 3.92 9.99 1.07 4527388 0.00 0.00 >> >> >>>> Curl_hash_pick >> >> >>>> 3.43 10.93 0.94 >> >> >>>> locking_function >> >> >>>> 3.04 11.76 0.83 909759 0.00 0.00 >> >> >>>> curl_multi_socket_action >> >> >>>> 3.00 12.58 0.82 97252 0.00 0.00 >> >> >>>> Curl_hash_clean_with_criterium >> >> >>>> 2.71 13.32 0.74 179704016 0.00 0.00 >> >> >>>> Curl_raw_toupper >> >> >>>> 2.68 14.05 0.73 4646524 0.00 0.00 >> >> >>>> multi_runsingle >> >> >>>> 2.44 14.71 0.67 13862992 0.00 0.00 >> >> >>>> client_tracing_function >> >> >>>> 2.42 15.37 0.66 319735 0.00 0.00 >> >> >>>> curl_multi_remove_handle >> >> >>>> 2.42 16.03 0.66 22 0.03 0.03 >> >> >>>> ares__init_list_node >> >> >>>> 1.94 16.56 0.53 9930244 0.00 0.00 >> >> >>>> hostcache_timestamp_remove >> >> >>>> 1.94 17.09 0.53 169320088 0.00 0.00 addbyter >> >> >>>> 1.80 17.58 0.49 id_function >> >> >>>> 1.36 17.95 0.37 7057146 0.00 0.00 >> >> >>>> Curl_pgrsUpdate >> >> >>>> 1.36 18.32 0.37 16835 0.00 0.00 >> >> >>>> curl_multi_socket_all >> >> >>>> 1.25 18.66 0.34 16719 0.00 0.00 >> >> >>>> curl_multi_perform >> >> >>>> 1.21 18.99 0.33 329403 0.00 0.00 >> >> >>>> Curl_http_readwrite_headers >> >> >>>> 1.17 19.31 0.32 4036624 0.00 0.00 Curl_splay >> >> >>>> 1.14 19.62 0.31 20838902 0.00 0.00 >> >> >>>> Curl_raw_nequal >> >> >>>> 0.82 19.85 0.23 2764123 0.00 0.00 Curl_infof >> >> >>>> 0.81 20.07 0.22 5933857 0.00 0.00 ossl_recv >> >> >>>> 0.77 20.28 0.21 323112 0.00 0.00 >> >> >>>> Curl_splayremovebyaddr >> >> >>>> 0.75 20.48 0.21 5943991 0.00 0.00 Curl_read >> >> >>>> 0.70 20.67 0.19 425174 0.00 0.00 event_del >> >> >>>> 0.70 20.86 0.19 98652 0.00 0.00 >> >> >>>> Curl_if_is_interface_name >> >> >>>> 0.70 21.05 0.19 8677733 0.00 0.00 >> >> >>>> Curl_socket_check >> >> >>>> 0.59 21.21 0.16 4129554 0.00 0.00 >> >> >>>> fd_key_compare >> >> >>>> 0.57 21.37 0.16 14017019 0.00 0.00 Curl_debug >> >> >>>> 0.55 21.52 0.15 9376861 0.00 0.00 >> >> >>>> stat_data_in_add >> >> >>>> 0.55 21.67 0.15 6840468 0.00 0.00 >> >> >>>> Curl_timeleft >> >> >>>> 0.55 21.82 0.15 2054602 0.00 0.00 >> >> >>>> Curl_raw_equal >> >> >>>> 0.55 21.97 0.15 1291611 0.00 0.00 Curl_expire >> >> >>>> 0.51 22.11 0.14 16570063 0.00 0.00 curlx_tvnow >> >> >>>> 0.51 22.25 0.14 13863284 0.00 0.00 >> >> >>>> scan_response >> >> >>>> 0.51 22.39 0.14 5498864 0.00 0.00 Curl_setopt >> >> >>>> 0.48 22.52 0.13 97490291 0.00 0.00 >> >> >>>> curl_strequal >> >> >>>> 0.44 22.64 0.12 324253 0.00 0.00 Curl_http >> >> >>>> 0.44 22.76 0.12 394386 0.00 0.00 >> >> >>>> ossl_connect_common >> >> >>>> 0.40 22.87 0.11 3581335 0.00 0.00 Curl_getinfo >> >> >>>> 0.38 22.97 0.11 26983092 0.00 0.00 >> >> >>>> alloc_addbyter >> >> >>>> 0.37 23.07 0.10 8729044 0.00 0.00 >> >> >>>> Curl_client_write >> >> >>>> >> >> >>>> >> >> >>>> >> >> >>>> >> >> >>>> ------------------------------------------------------------------------------ >> >> >>>> Try before you buy = See our experts in action! >> >> >>>> The most comprehensive online learning library for Microsoft >> >> >>>> developers >> >> >>>> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, >> >> >>>> MVC3, >> >> >>>> Metro Style Apps, more. Free future releases when you subscribe >> >> >>>> now! >> >> >>>> http://p.sf.net/sfu/learndevnow-dev2 >> >> >>>> _______________________________________________ >> >> >>>> curl-loader-devel mailing list >> >> >>>> cur...@li... >> >> >>>> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel >> >> >>>> >> >> >>> >> >> >>> >> >> >>> >> >> >>> -- >> >> >>> Regards, >> >> >>> Robert Iakobashvili, Ph.D. >> >> >>> >> >> >>> Home: http://www.ghotit.com >> >> >>> Blog: http://www.ghotit.com/category/ghotit-blogs/ >> >> >>> Twitter: http://twitter.com/ghotit >> >> >>> Facebook: http://facebook.com/ghotit >> >> >>> >> >> >>> ...................................................................... >> >> >>> Ghotit Dyslexia >> >> >>> Assistive technology that understands you >> >> >>> >> >> >>> ...................................................................... >> >> >>> >> >> >>> >> >> >>> >> >> >>> >> >> >>> ------------------------------------------------------------------------------ >> >> >>> Try before you buy = See our experts in action! >> >> >>> The most comprehensive online learning library for Microsoft >> >> >>> developers >> >> >>> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, >> >> >>> MVC3, >> >> >>> Metro Style Apps, more. Free future releases when you subscribe >> >> >>> now! >> >> >>> http://p.sf.net/sfu/learndevnow-dev2 >> >> >>> _______________________________________________ >> >> >>> curl-loader-devel mailing list >> >> >>> cur...@li... >> >> >>> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel >> >> >>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> ------------------------------------------------------------------------------ >> >> >> Try before you buy = See our experts in action! >> >> >> The most comprehensive online learning library for Microsoft >> >> >> developers >> >> >> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, >> >> >> MVC3, >> >> >> Metro Style Apps, more. Free future releases when you subscribe now! >> >> >> http://p.sf.net/sfu/learndevnow-dev2 >> >> >> _______________________________________________ >> >> >> curl-loader-devel mailing list >> >> >> cur...@li... >> >> >> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel >> >> >> >> >> > >> >> > >> >> > >> >> > -- >> >> > Regards, >> >> > Robert Iakobashvili, Ph.D. >> >> > >> >> > Home: http://www.ghotit.com >> >> > Blog: http://www.ghotit.com/category/ghotit-blogs/ >> >> > Twitter: http://twitter.com/ghotit >> >> > Facebook: http://facebook.com/ghotit >> >> > >> >> > ...................................................................... >> >> > Ghotit Dyslexia >> >> > Assistive technology that understands you >> >> > >> >> > ...................................................................... >> >> >> >> >> >> >> >> -- >> >> Regards, >> >> Robert Iakobashvili, Ph.D. >> >> >> >> Home: http://www.ghotit.com >> >> Blog: http://www.ghotit.com/category/ghotit-blogs/ >> >> Twitter: http://twitter.com/ghotit >> >> Facebook: http://facebook.com/ghotit >> >> ...................................................................... >> >> Ghotit Dyslexia >> >> Assistive technology that understands you >> >> ...................................................................... >> >> >> >> >> >> >> >> ------------------------------------------------------------------------------ >> >> Try before you buy = See our experts in action! >> >> The most comprehensive online learning library for Microsoft developers >> >> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, >> >> MVC3, >> >> Metro Style Apps, more. Free future releases when you subscribe now! >> >> http://p.sf.net/sfu/learndevnow-dev2 >> >> _______________________________________________ >> >> curl-loader-devel mailing list >> >> cur...@li... >> >> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel >> > >> > >> > >> > >> > ------------------------------------------------------------------------------ >> > Keep Your Developer Skills Current with LearnDevNow! >> > The most comprehensive online learning library for Microsoft developers >> > is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, >> > Metro Style Apps, more. Free future releases when you subscribe now! >> > http://p.sf.net/sfu/learndevnow-d2d >> > _______________________________________________ >> > curl-loader-devel mailing list >> > cur...@li... >> > https://lists.sourceforge.net/lists/listinfo/curl-loader-devel >> > >> >> >> >> -- >> Regards, >> Robert Iakobashvili, Ph.D. >> >> Home: http://www.ghotit.com >> Blog: http://www.ghotit.com/category/ghotit-blogs/ >> Twitter: http://twitter.com/ghotit >> Facebook: http://facebook.com/ghotit >> ...................................................................... >> Ghotit Dyslexia >> Assistive technology that understands you >> ...................................................................... >> >> >> ------------------------------------------------------------------------------ >> Keep Your Developer Skills Current with LearnDevNow! >> The most comprehensive online learning library for Microsoft developers >> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, >> Metro Style Apps, more. Free future releases when you subscribe now! >> http://p.sf.net/sfu/learndevnow-d2d >> _______________________________________________ >> curl-loader-devel mailing list >> cur...@li... >> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel > > > > ------------------------------------------------------------------------------ > Keep Your Developer Skills Current with LearnDevNow! > The most comprehensive online learning library for Microsoft developers > is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, > Metro Style Apps, more. Free future releases when you subscribe now! > http://p.sf.net/sfu/learndevnow-d2d > _______________________________________________ > curl-loader-devel mailing list > cur...@li... > https://lists.sourceforge.net/lists/listinfo/curl-loader-devel > -- Regards, Robert Iakobashvili, Ph.D. Home: http://www.ghotit.com Blog: http://www.ghotit.com/category/ghotit-blogs/ Twitter: http://twitter.com/ghotit Facebook: http://facebook.com/ghotit ...................................................................... Ghotit Dyslexia Assistive technology that understands you ...................................................................... |
|
From: Fred H. <di...@gm...> - 2012-03-06 16:27:00
|
different order if use HTTP URLs: # gprof /usr/bin/curl-loader gmon.out -p | head -20 Flat profile: Each sample counts as 0.01 seconds. % cumulative self self total time seconds seconds calls s/call s/call name 39.57 154.26 154.26 92591 0.00 0.00 Curl_hash_clean_with_criterium 20.06 232.47 78.21 4286341 0.00 0.00 Curl_hash_pick 16.07 295.12 62.65 253498 0.00 0.00 Curl_hash_add 12.55 344.06 48.93 1073699488 0.00 0.00 * hostcache_timestamp_remove* 6.89 370.94 26.88 361026319 0.00 0.00 Curl_str_key_compare 0.38 372.43 1.49 1153627 0.00 0.00 dprintf_formatf 0.37 373.86 1.44 2007540 0.00 0.00 Curl_readwrite 0.31 375.08 1.22 85075 0.00 0.00 Curl_num_addresses 0.25 376.04 0.96 3092327 0.00 0.00 multi_runsingle 0.23 376.92 0.88 263320355 0.00 0.00 ares__is_list_empty 0.20 377.71 0.79 39752 0.00 0.01 curl_multi_socket_all 0.19 378.44 0.73 256897 0.00 0.00 Curl_hash_str 0.17 379.11 0.67 89266 0.00 0.00 ares_cancel 0.15 379.70 0.59 90755 0.00 0.00 Curl_if2ip 0.13 380.21 0.51 3191375 0.00 0.00 client_tracing_function 2012/3/7 Fred Huang <di...@gm...> > 1. the major CPU consumer is curl-loader. here is the top: > top - 07:36:36 up 23:31, 5 users, load average: 4.32, 2.26, 0.91 > Tasks: 237 total, 2 running, 235 sleeping, 0 stopped, 0 zombie > Cpu0 : 76.6%us, 3.6%sy, 0.0%ni, 19.4%id, 0.0%wa, 0.0%hi, 0.3%si, > 0.0% > Cpu1 : 72.4%us, 4.0%sy, 0.0%ni, 22.9%id, 0.0%wa, 0.0%hi, 0.7%si, > 0.0% > Cpu2 : 65.8%us, 3.3%sy, 0.0%ni, 30.6%id, 0.0%wa, 0.0%hi, 0.3%si, > 0.0% > Cpu3 : 75.2%us, 4.3%sy, 0.0%ni, 19.5%id, 0.0%wa, 0.0%hi, 1.0%si, > 0.0% > Cpu4 : 79.1%us, 4.0%sy, 0.0%ni, 16.8%id, 0.0%wa, 0.0%hi, 0.0%si, > 0.0% > Cpu5 : 75.8%us, 4.3%sy, 0.0%ni, 19.2%id, 0.0%wa, 0.0%hi, 0.7%si, > 0.0% > Cpu6 : 74.1%us, 5.0%sy, 0.0%ni, 20.3%id, 0.0%wa, 0.0%hi, 0.7%si, > 0.0% > Cpu7 : 78.7%us, 4.0%sy, 0.0%ni, 16.6%id, 0.0%wa, 0.0%hi, 0.7%si, > 0.0% > Mem: 16209932k total, 3829500k used, 12380432k free, 99112k buffers > Swap: 18448376k total, 0k used, 18448376k free, 320284k cached > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 30795 root 20 0 3200m 2.8g 2296 S *79.2* 18.4 21:26.84 curl-loader > 1832 nobody 20 0 13944 1736 580 S 0.1 0.0 0:32.65 dnsmasq > 9 root 20 0 0 0 0 S 0.0 0.0 0:00.53 ksoftirqd/1 > 30825 root 20 0 15220 1360 960 R 0.0 0.0 0:00.42 top > 1 root 20 0 19400 1568 1256 S 0.0 0.0 0:00.75 init > 2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd > 3 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/0 > 4 root 20 0 0 0 0 S 0.0 0.0 0:00.06 ksoftirqd/0 > 5 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/0 > > > 2. 1000 loading clients > > 3. yes, tried 0.52 but no lucky -- CAPS 380, throughput 200Mbps, CPU 90% > top - 08:17:10 up 1 day, 12 min, 8 users, load average: 4.66, 3.39, 1.88 > Tasks: 245 total, 2 running, 243 sleeping, 0 stopped, 0 zombie > Cpu0 : 82.2%us, 4.0%sy, 0.0%ni, 13.9%id, 0.0%wa, 0.0%hi, 0.0%si, > 0.0% > Cpu1 : 86.1%us, 4.3%sy, 0.0%ni, 7.9%id, 0.0%wa, 0.0%hi, 1.7%si, > 0.0% > Cpu2 : 85.3%us, 4.0%sy, 0.0%ni, 10.0%id, 0.0%wa, 0.0%hi, 0.7%si, > 0.0% > Cpu3 : 81.8%us, 3.0%sy, 0.0%ni, 14.2%id, 0.0%wa, 0.0%hi, 1.0%si, > 0.0% > Cpu4 : 83.7%us, 4.0%sy, 0.0%ni, 12.3%id, 0.0%wa, 0.0%hi, 0.0%si, > 0.0% > Cpu5 : 92.7%us, 3.3%sy, 0.0%ni, 3.3%id, 0.0%wa, 0.0%hi, 0.7%si, > 0.0% > Cpu6 : 89.7%us, 2.0%sy, 0.0%ni, 8.0%id, 0.0%wa, 0.0%hi, 0.3%si, > 0.0% > Cpu7 : 88.1%us, 3.6%sy, 0.0%ni, 6.6%id, 0.0%wa, 0.0%hi, 1.7%si, > 0.0% > Mem: 16209932k total, 1795748k used, 14414184k free, 102824k buffers > Swap: 18448376k total, 0k used, 18448376k free, 382988k cached > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 12236 root 20 0 1150m 894m 2572 S 90.7 5.7 51:11.91 curl-loader > 1832 nobody 20 0 13944 1736 580 S 0.1 0.0 0:47.04 dnsmasq > > ]# gprof /usr/bin/curl-loader gmon.out -p | head -20 > Flat profile: > Each sample counts as 0.01 seconds. > % cumulative self self total > time seconds seconds calls s/call s/call name > 43.92 151.95 151.95 157295 0.00 0.00 > Curl_hash_clean_with_criterium > 22.90 231.17 79.22 1712190971 0.00 0.00 > hostcache_timestamp_remove > 13.15 276.68 45.51 3260321 0.00 0.00 Curl_hash_pick > 7.51 302.65 25.97 394171 0.00 0.00 Curl_hash_add > 5.98 323.34 20.68 567437518 0.00 0.00 Curl_str_key_compare > 1.00 326.80 3.46 5380220 0.00 0.00 dprintf_formatf > 0.41 328.21 1.41 locking_function > 0.32 329.32 1.11 469222138 0.00 0.00 ares__is_list_empty > 0.31 330.40 1.08 3284508 0.00 0.00 multi_runsingle > 0.28 331.38 0.98 10006806 0.00 0.00 > client_tracing_function > 0.24 332.21 0.83 151951 0.00 0.00 Curl_num_addresses > 0.22 332.97 0.77 188075672 0.00 0.00 addbyter > 0.19 333.64 0.67 841840 0.00 0.00 Curl_readwrite > 0.19 334.29 0.65 id_function > 0.18 334.90 0.61 39843 0.00 0.01 curl_multi_socket_all > > > > 4. yes, it is correct for 0.56. If use same URL list by replacing https to > http, I can get CAPS 1500 and throughput 870Mbps with CPU% < 60% > > > 2012/3/6 Robert Iakobashvili <cor...@gm...> > >> Several questions: >> >> 1. Could you please run top and see if there are other processes burning >> CPU? >> 2. What is the number of loading clients that you use? >> 3. Could you try curl-loader-0.52 if it compiles for you? >> If not, please try to place curl-loader-7.19.7.tar.gz and >> c-ares-16.0.tar.gz, >> update their versions in Makefile and rebuild by make cleanall; make >> >> 4. Is it correct to say that for HTTP URLs (not HTTPS) you are getting >> a good throughput >> and CAPS with curl-loader-0.56? >> >> Thanks, >> Robert >> >> On Tue, Mar 6, 2012 at 3:52 AM, Fred Huang <di...@gm...> wrote: >> > TIMEOUT -1 -> 60 works a little better but no performance improvement on >> > CAPS or throughput. >> > >> > test 5: >> > enable c-ares; >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, 60); >> > >> > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve >> *.com to >> > one IP address >> > number of domain names in URLs: 780,000 >> > number of client: 1000 >> > CPU usage: 74% >> > cpu% irq% sirq% sys% iowt% mem_used buf&cached >> > 74.0 0.0 0.7 4.2 0.0 3530.6Mb 250.8Mb >> > SSL TPS: 360 >> > SSL throughput: 220Mbps >> > >> > # gprof /usr/bin/curl-loader gmon.out -p | head -50 >> > Flat profile: >> > Each sample counts as 0.01 seconds. >> > % cumulative self self total >> > time seconds seconds calls s/call s/call name >> > 26.63 23.13 23.13 119226 0.00 0.00 >> > Curl_hash_clean_with_criterium >> > 20.34 40.80 17.67 317792146 0.00 0.00 >> > hostcache_timestamp_remove >> > 11.27 50.59 9.79 2579903 0.00 0.00 Curl_hash_pick >> > 7.84 57.40 6.81 120606 0.00 0.00 create_conn >> > 5.65 62.31 4.91 119117401 0.00 0.00 >> Curl_str_key_compare >> > 4.98 66.64 4.33 329829 0.00 0.00 Curl_hash_add >> > 2.80 69.08 2.44 4476606 0.00 0.00 dprintf_formatf >> > 2.57 71.31 2.23 124019 0.00 0.00 ConnectionStore >> > 1.50 72.61 1.30 740997609 0.00 0.00 >> ares__is_list_empty >> > 1.16 73.61 1.01 locking_function >> > 0.76 74.27 0.66 238744 0.00 0.00 ares_cancel >> > 0.70 74.88 0.61 457564 0.00 0.00 >> > curl_multi_socket_action >> > 0.67 75.46 0.58 118616 0.00 0.00 >> > curl_multi_remove_handle >> > 0.58 75.97 0.51 8211502 0.00 0.00 >> > client_tracing_function >> > 0.53 76.43 0.46 733561 0.00 0.00 Curl_readwrite >> > 0.51 76.87 0.44 2183179 0.00 0.00 multi_runsingle >> > 0.51 77.31 0.44 166585567 0.00 0.00 addbyter >> > 0.44 77.69 0.38 2175805 0.00 0.00 Curl_raw_equal >> > 0.43 78.06 0.38 id_function >> > 0.40 78.41 0.35 29789346 0.00 0.00 Curl_socket_check >> > 0.40 78.76 0.35 479671 0.00 0.00 >> ossl_connect_common >> > 0.40 79.11 0.35 490639 0.00 0.00 >> ares__init_list_node >> > 0.33 79.40 0.29 94197670 0.00 0.00 Curl_raw_toupper >> > 0.33 79.69 0.29 3268031 0.00 0.00 Curl_pgrsUpdate >> > 0.28 79.93 0.24 8797 0.00 0.00 curl_multi_perform >> > 0.27 80.17 0.24 2233641 0.00 0.00 Curl_infof >> > 0.26 80.40 0.23 1839837 0.00 0.00 Curl_splay >> > 0.24 80.61 0.21 121713028 0.00 0.00 curl_strequal >> > 0.21 80.79 0.18 243333 0.00 0.00 >> Curl_ssl_getsessionid >> > 0.18 80.95 0.16 122180 0.00 0.00 Curl_num_addresses >> > 0.17 81.09 0.15 8271666 0.00 0.00 Curl_debug >> > 0.16 81.23 0.14 18902452 0.00 0.00 alloc_addbyter >> > 0.16 81.37 0.14 8851 0.00 0.00 >> curl_multi_socket_all >> > 0.14 81.50 0.13 2236769 0.00 0.00 Curl_read >> > 0.14 81.62 0.12 2271700 0.00 0.00 fd_key_compare >> > 0.14 81.74 0.12 323167 0.00 0.00 event_del >> > 0.14 81.86 0.12 205436 0.00 0.00 read_udp_packets >> > 0.14 81.98 0.12 200979 0.00 0.00 processfds >> > 0.13 82.09 0.11 123886 0.00 0.00 >> > Curl_if_is_interface_name >> > 0.13 82.20 0.11 120350 0.00 0.00 Curl_http >> > 0.13 82.31 0.11 ssl_tls_trace >> > 0.12 82.41 0.10 119229 0.00 0.00 >> Curl_splayremovebyaddr >> > 0.12 82.51 0.10 7798717 0.00 0.00 Curl_raw_nequal >> > 0.12 82.61 0.10 784485 0.00 0.00 event_add >> > 0.10 82.70 0.09 643482 0.00 0.00 Curl_expire >> > >> > >> > >> > >> > >> > test 6: >> > enable c-ares; >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, 60); >> > CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.23.0 >> > >> > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve >> *.com to >> > one IP address >> > number of domain names in URLs: 780,000 >> > number of client: 1000 >> > CPU usage: 86% >> > cpu% irq% sirq% sys% iowt% mem_used buf&cached >> > 85.9 0.0 0.2 2.1 0.0 3537.2Mb 261.4Mb >> > SSL TPS: 360 >> > SSL throughput: 200Mbps >> > >> > # gprof /usr/bin/curl-loader gmon.out -p | head -20 >> > Flat profile: >> > Each sample counts as 0.01 seconds. >> > % cumulative self self total >> > time seconds seconds calls s/call s/call name >> > 27.19 19.96 19.96 103472 0.00 0.00 >> > Curl_hash_clean_with_criterium >> > 21.73 35.91 15.95 274168981 0.00 0.00 >> > hostcache_timestamp_remove >> > 10.33 43.49 7.58 2256441 0.00 0.00 Curl_hash_pick >> > 7.21 48.78 5.29 104540 0.00 0.00 create_conn >> > 6.20 53.33 4.55 103413405 0.00 0.00 >> Curl_str_key_compare >> > 4.88 56.91 3.58 287109 0.00 0.00 Curl_hash_add >> > 3.02 59.13 2.22 3797855 0.00 0.00 dprintf_formatf >> > 2.10 60.67 1.54 107711 0.00 0.00 ConnectionStore >> > 1.65 61.88 1.21 642725327 0.00 0.00 >> ares__is_list_empty >> > 1.19 62.76 0.88 locking_function >> > 0.87 63.40 0.64 207817 0.00 0.00 ares_cancel >> > 0.56 63.81 0.41 404108 0.00 0.00 >> > curl_multi_socket_action >> > 0.54 64.21 0.40 425603 0.00 0.00 >> ares__init_list_node >> > 0.53 64.60 0.39 7068112 0.00 0.00 >> > client_tracing_function >> > 0.49 64.96 0.36 1908587 0.00 0.00 multi_runsingle >> > >> > >> > >> > test 7: >> > enable c-ares; >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, -1); >> > CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.23.0 >> > >> > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve >> *.com to >> > one IP address >> > number of domain names in URLs: 780,000 >> > number of client: 1000 >> > CPU usage: 86% >> > cpu% irq% sirq% sys% iowt% mem_used buf&cached >> > 86.6 0.0 0.6 3.3 0.0 3624.8Mb 263.0Mb >> > SSL TPS: 300 >> > SSL throughput: 200Mbps >> > >> > # gprof /usr/bin/curl-loader gmon.out -p | head -20 >> > Flat profile: >> > Each sample counts as 0.01 seconds. >> > % cumulative self self total >> > time seconds seconds calls s/call s/call name >> > 39.83 369.71 369.71 207173 0.00 0.00 >> > Curl_hash_clean_with_criterium >> > 24.27 594.98 225.27 3198337733 0.00 0.00 >> > hostcache_timestamp_remove >> > 13.75 722.59 127.61 4312864 0.00 0.00 Curl_hash_pick >> > 9.19 807.87 85.28 546507 0.00 0.00 Curl_hash_add >> > 6.14 864.88 57.01 1022419684 0.00 0.00 >> Curl_str_key_compare >> > 2.06 884.00 19.12 203749 0.00 0.00 create_conn >> > 0.54 888.99 4.99 213306 0.00 0.00 ConnectionStore >> > 0.49 893.50 4.51 7331251 0.00 0.00 dprintf_formatf >> > 0.29 896.16 2.66 1287182329 0.00 0.00 >> ares__is_list_empty >> > 0.23 898.28 2.12 locking_function >> > 0.20 900.10 1.82 212210 0.00 0.00 Curl_num_addresses >> > 0.17 901.66 1.56 198385 0.00 0.00 >> > curl_multi_remove_handle >> > 0.12 902.81 1.15 411114 0.00 0.00 ares_cancel >> > 0.12 903.95 1.14 3683448 0.00 0.00 multi_runsingle >> > 0.11 904.97 1.03 13664120 0.00 0.00 >> > client_tracing_function >> > >> > >> > test 8: >> > enable c-ares; >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, 60); >> > CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.24.0 >> > >> > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve >> *.com to >> > one IP address >> > number of domain names in URLs: 780,000 >> > number of client: 1000 >> > CPU usage: 75% >> > cpu% irq% sirq% sys% iowt% mem_used buf&cached >> > 75.2 0.0 0.2 2.1 0.0 3537.2Mb 261.4Mb >> > SSL TPS: 380 >> > SSL throughput: 220Mbps >> > >> > # gprof /usr/bin/curl-loader gmon.out -p | head -20 >> > Flat profile: >> > Each sample counts as 0.01 seconds. >> > % cumulative self self total >> > time seconds seconds calls s/call s/call name >> > 27.58 12.31 12.31 71140 0.00 0.00 >> > Curl_hash_clean_with_criterium >> > 19.09 20.83 8.52 176729692 0.00 0.00 >> > hostcache_timestamp_remove >> > 10.44 25.49 4.66 1589309 0.00 0.00 Curl_hash_pick >> > 6.85 28.55 3.06 73204 0.00 0.00 create_conn >> > 5.56 31.03 2.48 67620530 0.00 0.00 >> Curl_str_key_compare >> > 4.41 33.00 1.97 200684 0.00 0.00 Curl_hash_add >> > 3.56 34.59 1.59 2717956 0.00 0.00 dprintf_formatf >> > 1.72 35.36 0.77 75247 0.00 0.00 ConnectionStore >> > 1.70 36.12 0.76 444342256 0.00 0.00 >> ares__is_list_empty >> > 1.23 36.67 0.55 locking_function >> > 1.16 37.19 0.52 143342 0.00 0.00 ares_cancel >> > 0.99 37.63 0.44 72211 0.00 0.00 >> > curl_multi_remove_handle >> > 0.74 37.96 0.33 1333371 0.00 0.00 multi_runsingle >> > 0.68 38.27 0.31 101147798 0.00 0.00 addbyter >> > 0.58 38.53 0.26 288737 0.00 0.00 >> > curl_multi_socket_action >> > >> > >> > >> > test 9: >> > enable c-ares; >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, -1); >> > CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.24.0 >> > >> > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve >> *.com to >> > one IP address >> > number of domain names in URLs: 780,000 >> > number of client: 1000 >> > CPU usage: 75% >> > cpu% irq% sirq% sys% iowt% mem_used buf&cached >> > 75.1 0.0 0.6 4.1 0.0 3545.2Mb 260.1Mb >> > SSL TPS: 380 >> > SSL throughput: 200Mbps >> > >> > # gprof /usr/bin/curl-loader gmon.out -p | head -20 >> > Flat profile: >> > Each sample counts as 0.01 seconds. >> > % cumulative self self total >> > time seconds seconds calls s/call s/call name >> > 34.00 31.13 31.13 72495 0.00 0.00 >> > Curl_hash_clean_with_criterium >> > 23.64 52.78 21.65 386070412 0.00 0.00 >> > hostcache_timestamp_remove >> > 13.28 64.94 12.16 1619440 0.00 0.00 Curl_hash_pick >> > 5.92 70.36 5.42 137653129 0.00 0.00 >> Curl_str_key_compare >> > 5.50 75.40 5.04 203018 0.00 0.00 Curl_hash_add >> > 4.55 79.57 4.17 74618 0.00 0.00 create_conn >> > 1.70 81.13 1.56 2757148 0.00 0.00 dprintf_formatf >> > 0.98 82.03 0.90 76945 0.00 0.00 ConnectionStore >> > 0.95 82.90 0.87 455775100 0.00 0.00 >> ares__is_list_empty >> > 0.57 83.42 0.53 locking_function >> > 0.39 83.78 0.36 146742 0.00 0.00 ares_cancel >> > 0.35 84.10 0.32 102968806 0.00 0.00 addbyter >> > 0.35 84.42 0.32 456587 0.00 0.00 Curl_readwrite >> > 0.35 84.74 0.32 73133 0.00 0.00 >> > curl_multi_remove_handle >> > 0.34 85.05 0.31 1360545 0.00 0.00 multi_runsingle >> > >> > >> > 2012/3/6 Robert Iakobashvili <cor...@gm...> >> >> >> >> If it will not be helpful this means that recent >> >> changes in libcurl library have broken DNS interface. >> >> >> >> You can try a previos version and the latest libcurl, like: >> >> >> >> http://curl.haxx.se/download/curl-7.23.0.tar.gz >> >> and >> >> http://curl.haxx.se/downloadcurl-7.24.0.tar.gz >> >> >> >> by: >> >> >> >> 1. downloading a libcurl version and placing it to >> >> curl-loader-0.56/packages ; >> >> 2. correcting the Makefile in curl-loader-0.56 >> >> CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.24 or CURL_VER:=7.23 >> >> 3. make cleanall; make >> >> >> >> Please, send us your observations. >> >> Thanks! >> >> Robert >> >> >> >> On Mon, Mar 5, 2012 at 6:57 PM, Robert Iakobashvili < >> cor...@gm...> >> >> wrote: >> >> > If so, please try to change >> >> > >> >> > loader.c, line 578 which looks like: >> >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, -1); >> >> > >> >> > 1. instead of -1 try to set some value like 60 >> >> > 2. make clean; make >> >> > >> >> > Thanks, >> >> > Robert >> >> > >> >> > On Mon, Mar 5, 2012 at 6:19 PM, Fred Huang <di...@gm...> wrote: >> >> >> >> >> >> Looks like the bottleneck comes from libcurl other than c-ares... >> >> >> test 3 (no ares patch applied): >> >> >> dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve >> *.com >> >> >> to one IP address >> >> >> number of domain names in URLs: 780,000 >> >> >> number of client: 1000 >> >> >> CPU usage: 85% >> >> >> cpu% irq% sirq% sys% iowt% mem_used buf&cached >> >> >> 85.6 0.0 0.8 5.9 0.0 3334.3Mb 112.0Mb >> >> >> SSL TPS: 320 >> >> >> SSL throughput: 200Mbps >> >> >> >> >> >> # gprof /usr/bin/curl-loader gmon.out -p | head -50 >> >> >> Flat profile: >> >> >> Each sample counts as 0.01 seconds. >> >> >> % cumulative self self total >> >> >> time seconds seconds calls s/call s/call name >> >> >> 40.80 120.16 120.16 116079 0.00 0.00 >> >> >> Curl_hash_clean_with_criterium >> >> >> 24.46 192.18 72.02 983166323 0.00 0.00 >> >> >> hostcache_timestamp_remove >> >> >> 13.30 231.35 39.17 2059341 0.00 0.00 Curl_hash_pick >> >> >> 6.45 250.35 19.00 240087 0.00 0.00 Curl_hash_add >> >> >> 4.81 264.51 14.16 334257102 0.00 0.00 >> >> >> Curl_str_key_compare >> >> >> 3.40 274.52 10.01 119514 0.00 0.00 create_conn >> >> >> 0.93 277.25 2.73 4478910 0.00 0.00 >> dprintf_formatf >> >> >> 0.90 279.91 2.66 122209 0.00 0.00 >> ConnectionStore >> >> >> 0.40 281.09 1.18 >> locking_function >> >> >> 0.27 281.88 0.79 114759 0.00 0.00 >> >> >> curl_multi_remove_handle >> >> >> 0.21 282.51 0.63 7993664 0.00 0.00 >> >> >> client_tracing_function >> >> >> 0.21 283.12 0.61 122685 0.00 0.00 >> >> >> Curl_num_addresses >> >> >> 0.19 283.68 0.56 397028 0.00 0.00 >> >> >> curl_multi_socket_action >> >> >> 0.19 284.23 0.55 2036116 0.00 0.00 >> multi_runsingle >> >> >> 0.17 284.72 0.49 703687 0.00 0.00 Curl_readwrite >> >> >> 0.16 285.19 0.47 id_function >> >> >> 0.15 285.62 0.43 29692029 0.00 0.00 >> Curl_socket_check >> >> >> 0.14 286.02 0.40 91973856 0.00 0.00 >> Curl_raw_toupper >> >> >> 0.13 286.40 0.38 164664144 0.00 0.00 addbyter >> >> >> 0.13 286.77 0.37 2180797 0.00 0.00 Curl_infof >> >> >> 0.11 287.10 0.33 122675 0.00 0.00 >> >> >> Curl_if_is_interface_name >> >> >> 0.10 287.39 0.29 471318 0.00 0.00 >> >> >> ossl_connect_common >> >> >> 0.10 287.67 0.28 2113528 0.00 0.00 Curl_raw_equal >> >> >> 0.09 287.93 0.26 8676 0.00 0.02 >> >> >> curl_multi_socket_all >> >> >> 0.08 288.17 0.24 1663262 0.00 0.00 Curl_splay >> >> >> 0.08 288.41 0.24 8505 0.00 0.01 >> >> >> curl_multi_perform >> >> >> 0.07 288.63 0.22 3114576 0.00 0.00 >> Curl_pgrsUpdate >> >> >> 0.06 288.80 0.17 119637 0.00 0.00 >> singleipconnect >> >> >> 0.06 288.97 0.17 7994818 0.00 0.00 Curl_debug >> >> >> 0.05 289.13 0.16 235820 0.00 0.00 event_del >> >> >> 0.05 289.29 0.16 119393071 0.00 0.00 curl_strequal >> >> >> 0.05 289.45 0.16 18535473 0.00 0.00 alloc_addbyter >> >> >> 0.05 289.61 0.16 356074 0.00 0.00 Curl_hash_str >> >> >> 0.05 289.76 0.15 1797233 0.00 0.00 fd_key_compare >> >> >> 0.05 289.91 0.15 234966 0.00 0.00 >> >> >> Curl_ssl_getsessionid >> >> >> 0.04 290.04 0.13 585853 0.00 0.00 >> >> >> Curl_llist_destroy >> >> >> 0.04 290.16 0.12 240826 0.00 0.00 Curl_resolv >> >> >> 0.03 290.26 0.10 7514328 0.00 0.00 >> Curl_raw_nequal >> >> >> 0.03 290.36 0.10 2214436 0.00 0.00 Curl_read >> >> >> 0.03 290.46 0.10 116872 0.00 0.00 >> >> >> Curl_http_readwrite_headers >> >> >> 0.03 290.56 0.10 ssl_tls_trace >> >> >> 0.03 290.66 0.10 8013097 0.00 0.00 scan_response >> >> >> 0.03 290.75 0.09 29488555 0.00 0.00 curlx_tvdiff >> >> >> 0.03 290.84 0.09 2142468 0.00 0.00 Curl_setopt >> >> >> 0.03 290.93 0.09 476299 0.00 0.00 Curl_expire >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> 2012/3/5 Robert Iakobashvili <cor...@gm...> >> >> >>> >> >> >>> Dear Fred Huang, >> >> >>> >> >> >>> Thanks for your posting. >> >> >>> >> >> >>> Could you please check if this is due to some badness at cares? >> >> >>> Thanks. >> >> >>> >> >> >>> Let's exclude usage of cares based DNS resolving by applying the >> patch >> >> >>> attached >> >> >>> and use instead global DNS cache. >> >> >>> >> >> >>> cd ./curl-loader-0.56 >> >> >>> patch -p1 < path-to/disable-cares-use-global-dns.patch >> >> >>> make cleanall; make >> >> >>> >> >> >>> See if it helps. >> >> >>> >> >> >>> Global cache, unfortunately, is not thread safe. >> >> >>> >> >> >>> If it helps, we'll try to restore the previous version of cares. >> >> >>> >> >> >>> Thanks, >> >> >>> Robert >> >> >>> >> >> >>> On Mon, Mar 5, 2012 at 4:28 PM, Fred Huang <di...@gm...> >> wrote: >> >> >>>> >> >> >>>> test 1: >> >> >>>> >> >> >>>> dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve >> >> >>>> *.com to one IP address >> >> >>>> number of domain names in all https URLs: 780,000 >> >> >>>> number of client: 1000 >> >> >>>> CPU usage: 78% >> >> >>>> cpu% irq% sirq% sys% iowt% mem_used buf&cached >> >> >>>> 78.8 0.0 0.6 3.8 0.0 3640.9Mb 282.1Mb >> >> >>>> SSL TPS: 300 >> >> >>>> SSL throughput: 200Mbps >> >> >>>> >> >> >>>> # gprof /usr/bin/curl-loader gmon.out -p | head -50 >> >> >>>> Flat profile: >> >> >>>> Each sample counts as 0.01 seconds. >> >> >>>> % cumulative self self total >> >> >>>> time seconds seconds calls s/call s/call name >> >> >>>> 38.55 127.10 127.10 130975 0.00 0.00 >> >> >>>> Curl_hash_clean_with_criterium >> >> >>>> 24.63 208.32 81.22 1247528407 0.00 0.00 >> >> >>>> hostcache_timestamp_remove >> >> >>>> 13.08 251.44 43.12 2791988 0.00 0.00 >> Curl_hash_pick >> >> >>>> 7.30 275.51 24.07 352490 0.00 0.00 >> Curl_hash_add >> >> >>>> 6.01 295.31 19.80 410760770 0.00 0.00 >> >> >>>> Curl_str_key_compare >> >> >>>> 2.95 305.04 9.73 131015 0.00 0.00 create_conn >> >> >>>> 0.88 307.94 2.90 4841390 0.00 0.00 >> dprintf_formatf >> >> >>>> 0.74 310.38 2.44 136036 0.00 0.00 >> ConnectionStore >> >> >>>> 0.48 311.96 1.58 814604911 0.00 0.00 >> >> >>>> ares__is_list_empty >> >> >>>> 0.36 313.16 1.20 >> >> >>>> locking_function >> >> >>>> 0.27 314.06 0.90 261608 0.00 0.00 ares_cancel >> >> >>>> 0.22 314.79 0.73 127417 0.00 0.00 >> >> >>>> curl_multi_remove_handle >> >> >>>> 0.19 315.40 0.62 8906301 0.00 0.00 >> >> >>>> client_tracing_function >> >> >>>> 0.17 315.97 0.57 135305 0.00 0.00 >> >> >>>> Curl_num_addresses >> >> >>>> 0.17 316.54 0.57 181937907 0.00 0.00 addbyter >> >> >>>> 0.17 317.09 0.55 2374363 0.00 0.00 >> multi_runsingle >> >> >>>> 0.16 317.62 0.53 541954 0.00 0.00 >> >> >>>> ares__init_list_node >> >> >>>> 0.16 318.14 0.52 494412 0.00 0.00 >> >> >>>> curl_multi_socket_action >> >> >>>> 0.14 318.60 0.46 33578897 0.00 0.00 >> >> >>>> Curl_socket_check >> >> >>>> 0.13 319.04 0.44 102951052 0.00 0.00 >> >> >>>> Curl_raw_toupper >> >> >>>> 0.13 319.48 0.44 787558 0.00 0.00 >> Curl_readwrite >> >> >>>> 0.13 319.91 0.44 id_function >> >> >>>> 0.12 320.29 0.38 391390 0.00 0.00 >> Curl_hash_str >> >> >>>> 0.09 320.58 0.29 9216 0.00 0.00 >> >> >>>> curl_multi_perform >> >> >>>> 0.08 320.86 0.28 2350908 0.00 0.00 >> Curl_raw_equal >> >> >>>> 0.08 321.14 0.28 2019415 0.00 0.00 Curl_splay >> >> >>>> 0.08 321.40 0.26 520625 0.00 0.00 >> >> >>>> ossl_connect_common >> >> >>>> 0.08 321.65 0.25 3532992 0.00 0.00 >> Curl_pgrsUpdate >> >> >>>> 0.07 321.87 0.22 131778537 0.00 0.00 >> curl_strequal >> >> >>>> 0.05 322.05 0.18 8954118 0.00 0.00 >> scan_response >> >> >>>> 0.05 322.23 0.18 533644 0.00 0.00 >> >> >>>> ares_expand_name >> >> >>>> 0.05 322.40 0.17 2477345 0.00 0.00 >> fd_key_compare >> >> >>>> 0.05 322.57 0.17 2429135 0.00 0.00 Curl_infof >> >> >>>> 0.05 322.74 0.17 130966 0.00 0.00 >> singleipconnect >> >> >>>> 0.05 322.90 0.16 9292 0.00 0.00 >> >> >>>> curl_multi_socket_all >> >> >>>> 0.05 323.06 0.16 270137 0.00 0.00 >> >> >>>> ares__get_hostent >> >> >>>> 0.05 323.21 0.15 8943657 0.00 0.00 Curl_debug >> >> >>>> 0.04 323.35 0.14 262493 0.00 0.00 >> >> >>>> Curl_ssl_getsessionid >> >> >>>> 0.04 323.48 0.13 566006 0.00 0.00 >> socket_callback >> >> >>>> 0.04 323.61 0.13 33983864 0.00 0.00 curlx_tvdiff >> >> >>>> 0.04 323.74 0.13 132853 0.00 0.00 >> >> >>>> Curl_http_readwrite_headers >> >> >>>> 0.04 323.86 0.12 344032 0.00 0.00 epoll_del >> >> >>>> 0.04 323.98 0.12 223477 0.00 0.00 Curl_poll >> >> >>>> 0.04 324.10 0.12 8526280 0.00 0.00 >> Curl_raw_nequal >> >> >>>> 0.03 324.21 0.11 348839 0.00 0.00 event_del >> >> >>>> >> >> >>>> >> >> >>>> >> >> >>>> >> >> >>>> test 2: >> >> >>>> >> >> >>>> dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve >> >> >>>> *.com to one IP address >> >> >>>> number of domain names in all https URLs: 1 >> >> >>>> number of client: 1000 >> >> >>>> CPU usage: 75% >> >> >>>> SSL TPS: 1300 >> >> >>>> SSL throughput: 700Mbps >> >> >>>> >> >> >>>> # gprof /usr/bin/curl-loader gmon.out -p | head -50 >> >> >>>> Flat profile: >> >> >>>> Each sample counts as 0.01 seconds. >> >> >>>> % cumulative self self total >> >> >>>> time seconds seconds calls s/call s/call name >> >> >>>> 8.98 2.45 2.45 5127312 0.00 0.00 >> dprintf_formatf >> >> >>>> 8.90 4.88 2.43 1292796503 0.00 0.00 >> >> >>>> ares__is_list_empty >> >> >>>> 6.12 6.55 1.67 322461 0.00 0.00 create_conn >> >> >>>> 4.58 7.80 1.25 419872 0.00 0.00 ares_cancel >> >> >>>> 4.10 8.92 1.12 2100162 0.00 0.00 >> Curl_readwrite >> >> >>>> 3.92 9.99 1.07 4527388 0.00 0.00 >> Curl_hash_pick >> >> >>>> 3.43 10.93 0.94 >> >> >>>> locking_function >> >> >>>> 3.04 11.76 0.83 909759 0.00 0.00 >> >> >>>> curl_multi_socket_action >> >> >>>> 3.00 12.58 0.82 97252 0.00 0.00 >> >> >>>> Curl_hash_clean_with_criterium >> >> >>>> 2.71 13.32 0.74 179704016 0.00 0.00 >> >> >>>> Curl_raw_toupper >> >> >>>> 2.68 14.05 0.73 4646524 0.00 0.00 >> multi_runsingle >> >> >>>> 2.44 14.71 0.67 13862992 0.00 0.00 >> >> >>>> client_tracing_function >> >> >>>> 2.42 15.37 0.66 319735 0.00 0.00 >> >> >>>> curl_multi_remove_handle >> >> >>>> 2.42 16.03 0.66 22 0.03 0.03 >> >> >>>> ares__init_list_node >> >> >>>> 1.94 16.56 0.53 9930244 0.00 0.00 >> >> >>>> hostcache_timestamp_remove >> >> >>>> 1.94 17.09 0.53 169320088 0.00 0.00 addbyter >> >> >>>> 1.80 17.58 0.49 id_function >> >> >>>> 1.36 17.95 0.37 7057146 0.00 0.00 >> Curl_pgrsUpdate >> >> >>>> 1.36 18.32 0.37 16835 0.00 0.00 >> >> >>>> curl_multi_socket_all >> >> >>>> 1.25 18.66 0.34 16719 0.00 0.00 >> >> >>>> curl_multi_perform >> >> >>>> 1.21 18.99 0.33 329403 0.00 0.00 >> >> >>>> Curl_http_readwrite_headers >> >> >>>> 1.17 19.31 0.32 4036624 0.00 0.00 Curl_splay >> >> >>>> 1.14 19.62 0.31 20838902 0.00 0.00 >> Curl_raw_nequal >> >> >>>> 0.82 19.85 0.23 2764123 0.00 0.00 Curl_infof >> >> >>>> 0.81 20.07 0.22 5933857 0.00 0.00 ossl_recv >> >> >>>> 0.77 20.28 0.21 323112 0.00 0.00 >> >> >>>> Curl_splayremovebyaddr >> >> >>>> 0.75 20.48 0.21 5943991 0.00 0.00 Curl_read >> >> >>>> 0.70 20.67 0.19 425174 0.00 0.00 event_del >> >> >>>> 0.70 20.86 0.19 98652 0.00 0.00 >> >> >>>> Curl_if_is_interface_name >> >> >>>> 0.70 21.05 0.19 8677733 0.00 0.00 >> >> >>>> Curl_socket_check >> >> >>>> 0.59 21.21 0.16 4129554 0.00 0.00 >> fd_key_compare >> >> >>>> 0.57 21.37 0.16 14017019 0.00 0.00 Curl_debug >> >> >>>> 0.55 21.52 0.15 9376861 0.00 0.00 >> >> >>>> stat_data_in_add >> >> >>>> 0.55 21.67 0.15 6840468 0.00 0.00 >> Curl_timeleft >> >> >>>> 0.55 21.82 0.15 2054602 0.00 0.00 >> Curl_raw_equal >> >> >>>> 0.55 21.97 0.15 1291611 0.00 0.00 Curl_expire >> >> >>>> 0.51 22.11 0.14 16570063 0.00 0.00 curlx_tvnow >> >> >>>> 0.51 22.25 0.14 13863284 0.00 0.00 >> scan_response >> >> >>>> 0.51 22.39 0.14 5498864 0.00 0.00 Curl_setopt >> >> >>>> 0.48 22.52 0.13 97490291 0.00 0.00 >> curl_strequal >> >> >>>> 0.44 22.64 0.12 324253 0.00 0.00 Curl_http >> >> >>>> 0.44 22.76 0.12 394386 0.00 0.00 >> >> >>>> ossl_connect_common >> >> >>>> 0.40 22.87 0.11 3581335 0.00 0.00 Curl_getinfo >> >> >>>> 0.38 22.97 0.11 26983092 0.00 0.00 >> alloc_addbyter >> >> >>>> 0.37 23.07 0.10 8729044 0.00 0.00 >> >> >>>> Curl_client_write >> >> >>>> >> >> >>>> >> >> >>>> >> >> >>>> >> ------------------------------------------------------------------------------ >> >> >>>> Try before you buy = See our experts in action! >> >> >>>> The most comprehensive online learning library for Microsoft >> >> >>>> developers >> >> >>>> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, >> >> >>>> MVC3, >> >> >>>> Metro Style Apps, more. Free future releases when you subscribe >> now! >> >> >>>> http://p.sf.net/sfu/learndevnow-dev2 >> >> >>>> _______________________________________________ >> >> >>>> curl-loader-devel mailing list >> >> >>>> cur...@li... >> >> >>>> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel >> >> >>>> >> >> >>> >> >> >>> >> >> >>> >> >> >>> -- >> >> >>> Regards, >> >> >>> Robert Iakobashvili, Ph.D. >> >> >>> >> >> >>> Home: http://www.ghotit.com >> >> >>> Blog: http://www.ghotit.com/category/ghotit-blogs/ >> >> >>> Twitter: http://twitter.com/ghotit >> >> >>> Facebook: http://facebook.com/ghotit >> >> >>> >> ...................................................................... >> >> >>> Ghotit Dyslexia >> >> >>> Assistive technology that understands you >> >> >>> >> ...................................................................... >> >> >>> >> >> >>> >> >> >>> >> >> >>> >> ------------------------------------------------------------------------------ >> >> >>> Try before you buy = See our experts in action! >> >> >>> The most comprehensive online learning library for Microsoft >> >> >>> developers >> >> >>> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, >> >> >>> MVC3, >> >> >>> Metro Style Apps, more. Free future releases when you subscribe >> now! >> >> >>> http://p.sf.net/sfu/learndevnow-dev2 >> >> >>> _______________________________________________ >> >> >>> curl-loader-devel mailing list >> >> >>> cur...@li... >> >> >>> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel >> >> >>> >> >> >> >> >> >> >> >> >> >> >> >> >> ------------------------------------------------------------------------------ >> >> >> Try before you buy = See our experts in action! >> >> >> The most comprehensive online learning library for Microsoft >> developers >> >> >> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, >> >> >> MVC3, >> >> >> Metro Style Apps, more. Free future releases when you subscribe now! >> >> >> http://p.sf.net/sfu/learndevnow-dev2 >> >> >> _______________________________________________ >> >> >> curl-loader-devel mailing list >> >> >> cur...@li... >> >> >> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel >> >> >> >> >> > >> >> > >> >> > >> >> > -- >> >> > Regards, >> >> > Robert Iakobashvili, Ph.D. >> >> > >> >> > Home: http://www.ghotit.com >> >> > Blog: http://www.ghotit.com/category/ghotit-blogs/ >> >> > Twitter: http://twitter.com/ghotit >> >> > Facebook: http://facebook.com/ghotit >> >> > >> ...................................................................... >> >> > Ghotit Dyslexia >> >> > Assistive technology that understands you >> >> > >> ...................................................................... >> >> >> >> >> >> >> >> -- >> >> Regards, >> >> Robert Iakobashvili, Ph.D. >> >> >> >> Home: http://www.ghotit.com >> >> Blog: http://www.ghotit.com/category/ghotit-blogs/ >> >> Twitter: http://twitter.com/ghotit >> >> Facebook: http://facebook.com/ghotit >> >> ...................................................................... >> >> Ghotit Dyslexia >> >> Assistive technology that understands you >> >> ...................................................................... >> >> >> >> >> >> >> ------------------------------------------------------------------------------ >> >> Try before you buy = See our experts in action! >> >> The most comprehensive online learning library for Microsoft developers >> >> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, >> MVC3, >> >> Metro Style Apps, more. Free future releases when you subscribe now! >> >> http://p.sf.net/sfu/learndevnow-dev2 >> >> _______________________________________________ >> >> curl-loader-devel mailing list >> >> cur...@li... >> >> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel >> > >> > >> > >> > >> ------------------------------------------------------------------------------ >> > Keep Your Developer Skills Current with LearnDevNow! >> > The most comprehensive online learning library for Microsoft developers >> > is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, >> > Metro Style Apps, more. Free future releases when you subscribe now! >> > http://p.sf.net/sfu/learndevnow-d2d >> > _______________________________________________ >> > curl-loader-devel mailing list >> > cur...@li... >> > https://lists.sourceforge.net/lists/listinfo/curl-loader-devel >> > >> >> >> >> -- >> Regards, >> Robert Iakobashvili, Ph.D. >> >> Home: http://www.ghotit.com >> Blog: http://www.ghotit.com/category/ghotit-blogs/ >> Twitter: http://twitter.com/ghotit >> Facebook: http://facebook.com/ghotit >> ...................................................................... >> Ghotit Dyslexia >> Assistive technology that understands you >> ...................................................................... >> >> >> ------------------------------------------------------------------------------ >> Keep Your Developer Skills Current with LearnDevNow! >> The most comprehensive online learning library for Microsoft developers >> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, >> Metro Style Apps, more. Free future releases when you subscribe now! >> http://p.sf.net/sfu/learndevnow-d2d >> _______________________________________________ >> curl-loader-devel mailing list >> cur...@li... >> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel >> > > |
|
From: Fred H. <di...@gm...> - 2012-03-06 16:17:26
|
1. the major CPU consumer is curl-loader. here is the top:
top - 07:36:36 up 23:31, 5 users, load average: 4.32, 2.26, 0.91
Tasks: 237 total, 2 running, 235 sleeping, 0 stopped, 0 zombie
Cpu0 : 76.6%us, 3.6%sy, 0.0%ni, 19.4%id, 0.0%wa, 0.0%hi, 0.3%si, 0.0%
Cpu1 : 72.4%us, 4.0%sy, 0.0%ni, 22.9%id, 0.0%wa, 0.0%hi, 0.7%si, 0.0%
Cpu2 : 65.8%us, 3.3%sy, 0.0%ni, 30.6%id, 0.0%wa, 0.0%hi, 0.3%si, 0.0%
Cpu3 : 75.2%us, 4.3%sy, 0.0%ni, 19.5%id, 0.0%wa, 0.0%hi, 1.0%si, 0.0%
Cpu4 : 79.1%us, 4.0%sy, 0.0%ni, 16.8%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%
Cpu5 : 75.8%us, 4.3%sy, 0.0%ni, 19.2%id, 0.0%wa, 0.0%hi, 0.7%si, 0.0%
Cpu6 : 74.1%us, 5.0%sy, 0.0%ni, 20.3%id, 0.0%wa, 0.0%hi, 0.7%si, 0.0%
Cpu7 : 78.7%us, 4.0%sy, 0.0%ni, 16.6%id, 0.0%wa, 0.0%hi, 0.7%si, 0.0%
Mem: 16209932k total, 3829500k used, 12380432k free, 99112k buffers
Swap: 18448376k total, 0k used, 18448376k free, 320284k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
30795 root 20 0 3200m 2.8g 2296 S *79.2* 18.4 21:26.84 curl-loader
1832 nobody 20 0 13944 1736 580 S 0.1 0.0 0:32.65 dnsmasq
9 root 20 0 0 0 0 S 0.0 0.0 0:00.53 ksoftirqd/1
30825 root 20 0 15220 1360 960 R 0.0 0.0 0:00.42 top
1 root 20 0 19400 1568 1256 S 0.0 0.0 0:00.75 init
2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd
3 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/0
4 root 20 0 0 0 0 S 0.0 0.0 0:00.06 ksoftirqd/0
5 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/0
2. 1000 loading clients
3. yes, tried 0.52 but no lucky -- CAPS 380, throughput 200Mbps, CPU 90%
top - 08:17:10 up 1 day, 12 min, 8 users, load average: 4.66, 3.39, 1.88
Tasks: 245 total, 2 running, 243 sleeping, 0 stopped, 0 zombie
Cpu0 : 82.2%us, 4.0%sy, 0.0%ni, 13.9%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%
Cpu1 : 86.1%us, 4.3%sy, 0.0%ni, 7.9%id, 0.0%wa, 0.0%hi, 1.7%si, 0.0%
Cpu2 : 85.3%us, 4.0%sy, 0.0%ni, 10.0%id, 0.0%wa, 0.0%hi, 0.7%si, 0.0%
Cpu3 : 81.8%us, 3.0%sy, 0.0%ni, 14.2%id, 0.0%wa, 0.0%hi, 1.0%si, 0.0%
Cpu4 : 83.7%us, 4.0%sy, 0.0%ni, 12.3%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%
Cpu5 : 92.7%us, 3.3%sy, 0.0%ni, 3.3%id, 0.0%wa, 0.0%hi, 0.7%si, 0.0%
Cpu6 : 89.7%us, 2.0%sy, 0.0%ni, 8.0%id, 0.0%wa, 0.0%hi, 0.3%si, 0.0%
Cpu7 : 88.1%us, 3.6%sy, 0.0%ni, 6.6%id, 0.0%wa, 0.0%hi, 1.7%si, 0.0%
Mem: 16209932k total, 1795748k used, 14414184k free, 102824k buffers
Swap: 18448376k total, 0k used, 18448376k free, 382988k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
12236 root 20 0 1150m 894m 2572 S 90.7 5.7 51:11.91 curl-loader
1832 nobody 20 0 13944 1736 580 S 0.1 0.0 0:47.04 dnsmasq
]# gprof /usr/bin/curl-loader gmon.out -p | head -20
Flat profile:
Each sample counts as 0.01 seconds.
% cumulative self self total
time seconds seconds calls s/call s/call name
43.92 151.95 151.95 157295 0.00 0.00
Curl_hash_clean_with_criterium
22.90 231.17 79.22 1712190971 0.00 0.00
hostcache_timestamp_remove
13.15 276.68 45.51 3260321 0.00 0.00 Curl_hash_pick
7.51 302.65 25.97 394171 0.00 0.00 Curl_hash_add
5.98 323.34 20.68 567437518 0.00 0.00 Curl_str_key_compare
1.00 326.80 3.46 5380220 0.00 0.00 dprintf_formatf
0.41 328.21 1.41 locking_function
0.32 329.32 1.11 469222138 0.00 0.00 ares__is_list_empty
0.31 330.40 1.08 3284508 0.00 0.00 multi_runsingle
0.28 331.38 0.98 10006806 0.00 0.00
client_tracing_function
0.24 332.21 0.83 151951 0.00 0.00 Curl_num_addresses
0.22 332.97 0.77 188075672 0.00 0.00 addbyter
0.19 333.64 0.67 841840 0.00 0.00 Curl_readwrite
0.19 334.29 0.65 id_function
0.18 334.90 0.61 39843 0.00 0.01 curl_multi_socket_all
4. yes, it is correct for 0.56. If use same URL list by replacing https to
http, I can get CAPS 1500 and throughput 870Mbps with CPU% < 60%
2012/3/6 Robert Iakobashvili <cor...@gm...>
> Several questions:
>
> 1. Could you please run top and see if there are other processes burning
> CPU?
> 2. What is the number of loading clients that you use?
> 3. Could you try curl-loader-0.52 if it compiles for you?
> If not, please try to place curl-loader-7.19.7.tar.gz and
> c-ares-16.0.tar.gz,
> update their versions in Makefile and rebuild by make cleanall; make
>
> 4. Is it correct to say that for HTTP URLs (not HTTPS) you are getting
> a good throughput
> and CAPS with curl-loader-0.56?
>
> Thanks,
> Robert
>
> On Tue, Mar 6, 2012 at 3:52 AM, Fred Huang <di...@gm...> wrote:
> > TIMEOUT -1 -> 60 works a little better but no performance improvement on
> > CAPS or throughput.
> >
> > test 5:
> > enable c-ares;
> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, 60);
> >
> > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve *.com
> to
> > one IP address
> > number of domain names in URLs: 780,000
> > number of client: 1000
> > CPU usage: 74%
> > cpu% irq% sirq% sys% iowt% mem_used buf&cached
> > 74.0 0.0 0.7 4.2 0.0 3530.6Mb 250.8Mb
> > SSL TPS: 360
> > SSL throughput: 220Mbps
> >
> > # gprof /usr/bin/curl-loader gmon.out -p | head -50
> > Flat profile:
> > Each sample counts as 0.01 seconds.
> > % cumulative self self total
> > time seconds seconds calls s/call s/call name
> > 26.63 23.13 23.13 119226 0.00 0.00
> > Curl_hash_clean_with_criterium
> > 20.34 40.80 17.67 317792146 0.00 0.00
> > hostcache_timestamp_remove
> > 11.27 50.59 9.79 2579903 0.00 0.00 Curl_hash_pick
> > 7.84 57.40 6.81 120606 0.00 0.00 create_conn
> > 5.65 62.31 4.91 119117401 0.00 0.00
> Curl_str_key_compare
> > 4.98 66.64 4.33 329829 0.00 0.00 Curl_hash_add
> > 2.80 69.08 2.44 4476606 0.00 0.00 dprintf_formatf
> > 2.57 71.31 2.23 124019 0.00 0.00 ConnectionStore
> > 1.50 72.61 1.30 740997609 0.00 0.00
> ares__is_list_empty
> > 1.16 73.61 1.01 locking_function
> > 0.76 74.27 0.66 238744 0.00 0.00 ares_cancel
> > 0.70 74.88 0.61 457564 0.00 0.00
> > curl_multi_socket_action
> > 0.67 75.46 0.58 118616 0.00 0.00
> > curl_multi_remove_handle
> > 0.58 75.97 0.51 8211502 0.00 0.00
> > client_tracing_function
> > 0.53 76.43 0.46 733561 0.00 0.00 Curl_readwrite
> > 0.51 76.87 0.44 2183179 0.00 0.00 multi_runsingle
> > 0.51 77.31 0.44 166585567 0.00 0.00 addbyter
> > 0.44 77.69 0.38 2175805 0.00 0.00 Curl_raw_equal
> > 0.43 78.06 0.38 id_function
> > 0.40 78.41 0.35 29789346 0.00 0.00 Curl_socket_check
> > 0.40 78.76 0.35 479671 0.00 0.00 ossl_connect_common
> > 0.40 79.11 0.35 490639 0.00 0.00
> ares__init_list_node
> > 0.33 79.40 0.29 94197670 0.00 0.00 Curl_raw_toupper
> > 0.33 79.69 0.29 3268031 0.00 0.00 Curl_pgrsUpdate
> > 0.28 79.93 0.24 8797 0.00 0.00 curl_multi_perform
> > 0.27 80.17 0.24 2233641 0.00 0.00 Curl_infof
> > 0.26 80.40 0.23 1839837 0.00 0.00 Curl_splay
> > 0.24 80.61 0.21 121713028 0.00 0.00 curl_strequal
> > 0.21 80.79 0.18 243333 0.00 0.00
> Curl_ssl_getsessionid
> > 0.18 80.95 0.16 122180 0.00 0.00 Curl_num_addresses
> > 0.17 81.09 0.15 8271666 0.00 0.00 Curl_debug
> > 0.16 81.23 0.14 18902452 0.00 0.00 alloc_addbyter
> > 0.16 81.37 0.14 8851 0.00 0.00
> curl_multi_socket_all
> > 0.14 81.50 0.13 2236769 0.00 0.00 Curl_read
> > 0.14 81.62 0.12 2271700 0.00 0.00 fd_key_compare
> > 0.14 81.74 0.12 323167 0.00 0.00 event_del
> > 0.14 81.86 0.12 205436 0.00 0.00 read_udp_packets
> > 0.14 81.98 0.12 200979 0.00 0.00 processfds
> > 0.13 82.09 0.11 123886 0.00 0.00
> > Curl_if_is_interface_name
> > 0.13 82.20 0.11 120350 0.00 0.00 Curl_http
> > 0.13 82.31 0.11 ssl_tls_trace
> > 0.12 82.41 0.10 119229 0.00 0.00
> Curl_splayremovebyaddr
> > 0.12 82.51 0.10 7798717 0.00 0.00 Curl_raw_nequal
> > 0.12 82.61 0.10 784485 0.00 0.00 event_add
> > 0.10 82.70 0.09 643482 0.00 0.00 Curl_expire
> >
> >
> >
> >
> >
> > test 6:
> > enable c-ares;
> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, 60);
> > CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.23.0
> >
> > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve *.com
> to
> > one IP address
> > number of domain names in URLs: 780,000
> > number of client: 1000
> > CPU usage: 86%
> > cpu% irq% sirq% sys% iowt% mem_used buf&cached
> > 85.9 0.0 0.2 2.1 0.0 3537.2Mb 261.4Mb
> > SSL TPS: 360
> > SSL throughput: 200Mbps
> >
> > # gprof /usr/bin/curl-loader gmon.out -p | head -20
> > Flat profile:
> > Each sample counts as 0.01 seconds.
> > % cumulative self self total
> > time seconds seconds calls s/call s/call name
> > 27.19 19.96 19.96 103472 0.00 0.00
> > Curl_hash_clean_with_criterium
> > 21.73 35.91 15.95 274168981 0.00 0.00
> > hostcache_timestamp_remove
> > 10.33 43.49 7.58 2256441 0.00 0.00 Curl_hash_pick
> > 7.21 48.78 5.29 104540 0.00 0.00 create_conn
> > 6.20 53.33 4.55 103413405 0.00 0.00
> Curl_str_key_compare
> > 4.88 56.91 3.58 287109 0.00 0.00 Curl_hash_add
> > 3.02 59.13 2.22 3797855 0.00 0.00 dprintf_formatf
> > 2.10 60.67 1.54 107711 0.00 0.00 ConnectionStore
> > 1.65 61.88 1.21 642725327 0.00 0.00
> ares__is_list_empty
> > 1.19 62.76 0.88 locking_function
> > 0.87 63.40 0.64 207817 0.00 0.00 ares_cancel
> > 0.56 63.81 0.41 404108 0.00 0.00
> > curl_multi_socket_action
> > 0.54 64.21 0.40 425603 0.00 0.00
> ares__init_list_node
> > 0.53 64.60 0.39 7068112 0.00 0.00
> > client_tracing_function
> > 0.49 64.96 0.36 1908587 0.00 0.00 multi_runsingle
> >
> >
> >
> > test 7:
> > enable c-ares;
> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, -1);
> > CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.23.0
> >
> > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve *.com
> to
> > one IP address
> > number of domain names in URLs: 780,000
> > number of client: 1000
> > CPU usage: 86%
> > cpu% irq% sirq% sys% iowt% mem_used buf&cached
> > 86.6 0.0 0.6 3.3 0.0 3624.8Mb 263.0Mb
> > SSL TPS: 300
> > SSL throughput: 200Mbps
> >
> > # gprof /usr/bin/curl-loader gmon.out -p | head -20
> > Flat profile:
> > Each sample counts as 0.01 seconds.
> > % cumulative self self total
> > time seconds seconds calls s/call s/call name
> > 39.83 369.71 369.71 207173 0.00 0.00
> > Curl_hash_clean_with_criterium
> > 24.27 594.98 225.27 3198337733 0.00 0.00
> > hostcache_timestamp_remove
> > 13.75 722.59 127.61 4312864 0.00 0.00 Curl_hash_pick
> > 9.19 807.87 85.28 546507 0.00 0.00 Curl_hash_add
> > 6.14 864.88 57.01 1022419684 0.00 0.00
> Curl_str_key_compare
> > 2.06 884.00 19.12 203749 0.00 0.00 create_conn
> > 0.54 888.99 4.99 213306 0.00 0.00 ConnectionStore
> > 0.49 893.50 4.51 7331251 0.00 0.00 dprintf_formatf
> > 0.29 896.16 2.66 1287182329 0.00 0.00
> ares__is_list_empty
> > 0.23 898.28 2.12 locking_function
> > 0.20 900.10 1.82 212210 0.00 0.00 Curl_num_addresses
> > 0.17 901.66 1.56 198385 0.00 0.00
> > curl_multi_remove_handle
> > 0.12 902.81 1.15 411114 0.00 0.00 ares_cancel
> > 0.12 903.95 1.14 3683448 0.00 0.00 multi_runsingle
> > 0.11 904.97 1.03 13664120 0.00 0.00
> > client_tracing_function
> >
> >
> > test 8:
> > enable c-ares;
> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, 60);
> > CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.24.0
> >
> > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve *.com
> to
> > one IP address
> > number of domain names in URLs: 780,000
> > number of client: 1000
> > CPU usage: 75%
> > cpu% irq% sirq% sys% iowt% mem_used buf&cached
> > 75.2 0.0 0.2 2.1 0.0 3537.2Mb 261.4Mb
> > SSL TPS: 380
> > SSL throughput: 220Mbps
> >
> > # gprof /usr/bin/curl-loader gmon.out -p | head -20
> > Flat profile:
> > Each sample counts as 0.01 seconds.
> > % cumulative self self total
> > time seconds seconds calls s/call s/call name
> > 27.58 12.31 12.31 71140 0.00 0.00
> > Curl_hash_clean_with_criterium
> > 19.09 20.83 8.52 176729692 0.00 0.00
> > hostcache_timestamp_remove
> > 10.44 25.49 4.66 1589309 0.00 0.00 Curl_hash_pick
> > 6.85 28.55 3.06 73204 0.00 0.00 create_conn
> > 5.56 31.03 2.48 67620530 0.00 0.00
> Curl_str_key_compare
> > 4.41 33.00 1.97 200684 0.00 0.00 Curl_hash_add
> > 3.56 34.59 1.59 2717956 0.00 0.00 dprintf_formatf
> > 1.72 35.36 0.77 75247 0.00 0.00 ConnectionStore
> > 1.70 36.12 0.76 444342256 0.00 0.00
> ares__is_list_empty
> > 1.23 36.67 0.55 locking_function
> > 1.16 37.19 0.52 143342 0.00 0.00 ares_cancel
> > 0.99 37.63 0.44 72211 0.00 0.00
> > curl_multi_remove_handle
> > 0.74 37.96 0.33 1333371 0.00 0.00 multi_runsingle
> > 0.68 38.27 0.31 101147798 0.00 0.00 addbyter
> > 0.58 38.53 0.26 288737 0.00 0.00
> > curl_multi_socket_action
> >
> >
> >
> > test 9:
> > enable c-ares;
> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, -1);
> > CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.24.0
> >
> > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve *.com
> to
> > one IP address
> > number of domain names in URLs: 780,000
> > number of client: 1000
> > CPU usage: 75%
> > cpu% irq% sirq% sys% iowt% mem_used buf&cached
> > 75.1 0.0 0.6 4.1 0.0 3545.2Mb 260.1Mb
> > SSL TPS: 380
> > SSL throughput: 200Mbps
> >
> > # gprof /usr/bin/curl-loader gmon.out -p | head -20
> > Flat profile:
> > Each sample counts as 0.01 seconds.
> > % cumulative self self total
> > time seconds seconds calls s/call s/call name
> > 34.00 31.13 31.13 72495 0.00 0.00
> > Curl_hash_clean_with_criterium
> > 23.64 52.78 21.65 386070412 0.00 0.00
> > hostcache_timestamp_remove
> > 13.28 64.94 12.16 1619440 0.00 0.00 Curl_hash_pick
> > 5.92 70.36 5.42 137653129 0.00 0.00
> Curl_str_key_compare
> > 5.50 75.40 5.04 203018 0.00 0.00 Curl_hash_add
> > 4.55 79.57 4.17 74618 0.00 0.00 create_conn
> > 1.70 81.13 1.56 2757148 0.00 0.00 dprintf_formatf
> > 0.98 82.03 0.90 76945 0.00 0.00 ConnectionStore
> > 0.95 82.90 0.87 455775100 0.00 0.00
> ares__is_list_empty
> > 0.57 83.42 0.53 locking_function
> > 0.39 83.78 0.36 146742 0.00 0.00 ares_cancel
> > 0.35 84.10 0.32 102968806 0.00 0.00 addbyter
> > 0.35 84.42 0.32 456587 0.00 0.00 Curl_readwrite
> > 0.35 84.74 0.32 73133 0.00 0.00
> > curl_multi_remove_handle
> > 0.34 85.05 0.31 1360545 0.00 0.00 multi_runsingle
> >
> >
> > 2012/3/6 Robert Iakobashvili <cor...@gm...>
> >>
> >> If it will not be helpful this means that recent
> >> changes in libcurl library have broken DNS interface.
> >>
> >> You can try a previos version and the latest libcurl, like:
> >>
> >> http://curl.haxx.se/download/curl-7.23.0.tar.gz
> >> and
> >> http://curl.haxx.se/downloadcurl-7.24.0.tar.gz
> >>
> >> by:
> >>
> >> 1. downloading a libcurl version and placing it to
> >> curl-loader-0.56/packages ;
> >> 2. correcting the Makefile in curl-loader-0.56
> >> CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.24 or CURL_VER:=7.23
> >> 3. make cleanall; make
> >>
> >> Please, send us your observations.
> >> Thanks!
> >> Robert
> >>
> >> On Mon, Mar 5, 2012 at 6:57 PM, Robert Iakobashvili <
> cor...@gm...>
> >> wrote:
> >> > If so, please try to change
> >> >
> >> > loader.c, line 578 which looks like:
> >> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, -1);
> >> >
> >> > 1. instead of -1 try to set some value like 60
> >> > 2. make clean; make
> >> >
> >> > Thanks,
> >> > Robert
> >> >
> >> > On Mon, Mar 5, 2012 at 6:19 PM, Fred Huang <di...@gm...> wrote:
> >> >>
> >> >> Looks like the bottleneck comes from libcurl other than c-ares...
> >> >> test 3 (no ares patch applied):
> >> >> dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve
> *.com
> >> >> to one IP address
> >> >> number of domain names in URLs: 780,000
> >> >> number of client: 1000
> >> >> CPU usage: 85%
> >> >> cpu% irq% sirq% sys% iowt% mem_used buf&cached
> >> >> 85.6 0.0 0.8 5.9 0.0 3334.3Mb 112.0Mb
> >> >> SSL TPS: 320
> >> >> SSL throughput: 200Mbps
> >> >>
> >> >> # gprof /usr/bin/curl-loader gmon.out -p | head -50
> >> >> Flat profile:
> >> >> Each sample counts as 0.01 seconds.
> >> >> % cumulative self self total
> >> >> time seconds seconds calls s/call s/call name
> >> >> 40.80 120.16 120.16 116079 0.00 0.00
> >> >> Curl_hash_clean_with_criterium
> >> >> 24.46 192.18 72.02 983166323 0.00 0.00
> >> >> hostcache_timestamp_remove
> >> >> 13.30 231.35 39.17 2059341 0.00 0.00 Curl_hash_pick
> >> >> 6.45 250.35 19.00 240087 0.00 0.00 Curl_hash_add
> >> >> 4.81 264.51 14.16 334257102 0.00 0.00
> >> >> Curl_str_key_compare
> >> >> 3.40 274.52 10.01 119514 0.00 0.00 create_conn
> >> >> 0.93 277.25 2.73 4478910 0.00 0.00 dprintf_formatf
> >> >> 0.90 279.91 2.66 122209 0.00 0.00 ConnectionStore
> >> >> 0.40 281.09 1.18
> locking_function
> >> >> 0.27 281.88 0.79 114759 0.00 0.00
> >> >> curl_multi_remove_handle
> >> >> 0.21 282.51 0.63 7993664 0.00 0.00
> >> >> client_tracing_function
> >> >> 0.21 283.12 0.61 122685 0.00 0.00
> >> >> Curl_num_addresses
> >> >> 0.19 283.68 0.56 397028 0.00 0.00
> >> >> curl_multi_socket_action
> >> >> 0.19 284.23 0.55 2036116 0.00 0.00 multi_runsingle
> >> >> 0.17 284.72 0.49 703687 0.00 0.00 Curl_readwrite
> >> >> 0.16 285.19 0.47 id_function
> >> >> 0.15 285.62 0.43 29692029 0.00 0.00
> Curl_socket_check
> >> >> 0.14 286.02 0.40 91973856 0.00 0.00
> Curl_raw_toupper
> >> >> 0.13 286.40 0.38 164664144 0.00 0.00 addbyter
> >> >> 0.13 286.77 0.37 2180797 0.00 0.00 Curl_infof
> >> >> 0.11 287.10 0.33 122675 0.00 0.00
> >> >> Curl_if_is_interface_name
> >> >> 0.10 287.39 0.29 471318 0.00 0.00
> >> >> ossl_connect_common
> >> >> 0.10 287.67 0.28 2113528 0.00 0.00 Curl_raw_equal
> >> >> 0.09 287.93 0.26 8676 0.00 0.02
> >> >> curl_multi_socket_all
> >> >> 0.08 288.17 0.24 1663262 0.00 0.00 Curl_splay
> >> >> 0.08 288.41 0.24 8505 0.00 0.01
> >> >> curl_multi_perform
> >> >> 0.07 288.63 0.22 3114576 0.00 0.00 Curl_pgrsUpdate
> >> >> 0.06 288.80 0.17 119637 0.00 0.00 singleipconnect
> >> >> 0.06 288.97 0.17 7994818 0.00 0.00 Curl_debug
> >> >> 0.05 289.13 0.16 235820 0.00 0.00 event_del
> >> >> 0.05 289.29 0.16 119393071 0.00 0.00 curl_strequal
> >> >> 0.05 289.45 0.16 18535473 0.00 0.00 alloc_addbyter
> >> >> 0.05 289.61 0.16 356074 0.00 0.00 Curl_hash_str
> >> >> 0.05 289.76 0.15 1797233 0.00 0.00 fd_key_compare
> >> >> 0.05 289.91 0.15 234966 0.00 0.00
> >> >> Curl_ssl_getsessionid
> >> >> 0.04 290.04 0.13 585853 0.00 0.00
> >> >> Curl_llist_destroy
> >> >> 0.04 290.16 0.12 240826 0.00 0.00 Curl_resolv
> >> >> 0.03 290.26 0.10 7514328 0.00 0.00 Curl_raw_nequal
> >> >> 0.03 290.36 0.10 2214436 0.00 0.00 Curl_read
> >> >> 0.03 290.46 0.10 116872 0.00 0.00
> >> >> Curl_http_readwrite_headers
> >> >> 0.03 290.56 0.10 ssl_tls_trace
> >> >> 0.03 290.66 0.10 8013097 0.00 0.00 scan_response
> >> >> 0.03 290.75 0.09 29488555 0.00 0.00 curlx_tvdiff
> >> >> 0.03 290.84 0.09 2142468 0.00 0.00 Curl_setopt
> >> >> 0.03 290.93 0.09 476299 0.00 0.00 Curl_expire
> >> >>
> >> >>
> >> >>
> >> >>
> >> >>
> >> >>
> >> >>
> >> >>
> >> >> 2012/3/5 Robert Iakobashvili <cor...@gm...>
> >> >>>
> >> >>> Dear Fred Huang,
> >> >>>
> >> >>> Thanks for your posting.
> >> >>>
> >> >>> Could you please check if this is due to some badness at cares?
> >> >>> Thanks.
> >> >>>
> >> >>> Let's exclude usage of cares based DNS resolving by applying the
> patch
> >> >>> attached
> >> >>> and use instead global DNS cache.
> >> >>>
> >> >>> cd ./curl-loader-0.56
> >> >>> patch -p1 < path-to/disable-cares-use-global-dns.patch
> >> >>> make cleanall; make
> >> >>>
> >> >>> See if it helps.
> >> >>>
> >> >>> Global cache, unfortunately, is not thread safe.
> >> >>>
> >> >>> If it helps, we'll try to restore the previous version of cares.
> >> >>>
> >> >>> Thanks,
> >> >>> Robert
> >> >>>
> >> >>> On Mon, Mar 5, 2012 at 4:28 PM, Fred Huang <di...@gm...>
> wrote:
> >> >>>>
> >> >>>> test 1:
> >> >>>>
> >> >>>> dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve
> >> >>>> *.com to one IP address
> >> >>>> number of domain names in all https URLs: 780,000
> >> >>>> number of client: 1000
> >> >>>> CPU usage: 78%
> >> >>>> cpu% irq% sirq% sys% iowt% mem_used buf&cached
> >> >>>> 78.8 0.0 0.6 3.8 0.0 3640.9Mb 282.1Mb
> >> >>>> SSL TPS: 300
> >> >>>> SSL throughput: 200Mbps
> >> >>>>
> >> >>>> # gprof /usr/bin/curl-loader gmon.out -p | head -50
> >> >>>> Flat profile:
> >> >>>> Each sample counts as 0.01 seconds.
> >> >>>> % cumulative self self total
> >> >>>> time seconds seconds calls s/call s/call name
> >> >>>> 38.55 127.10 127.10 130975 0.00 0.00
> >> >>>> Curl_hash_clean_with_criterium
> >> >>>> 24.63 208.32 81.22 1247528407 0.00 0.00
> >> >>>> hostcache_timestamp_remove
> >> >>>> 13.08 251.44 43.12 2791988 0.00 0.00
> Curl_hash_pick
> >> >>>> 7.30 275.51 24.07 352490 0.00 0.00 Curl_hash_add
> >> >>>> 6.01 295.31 19.80 410760770 0.00 0.00
> >> >>>> Curl_str_key_compare
> >> >>>> 2.95 305.04 9.73 131015 0.00 0.00 create_conn
> >> >>>> 0.88 307.94 2.90 4841390 0.00 0.00
> dprintf_formatf
> >> >>>> 0.74 310.38 2.44 136036 0.00 0.00
> ConnectionStore
> >> >>>> 0.48 311.96 1.58 814604911 0.00 0.00
> >> >>>> ares__is_list_empty
> >> >>>> 0.36 313.16 1.20
> >> >>>> locking_function
> >> >>>> 0.27 314.06 0.90 261608 0.00 0.00 ares_cancel
> >> >>>> 0.22 314.79 0.73 127417 0.00 0.00
> >> >>>> curl_multi_remove_handle
> >> >>>> 0.19 315.40 0.62 8906301 0.00 0.00
> >> >>>> client_tracing_function
> >> >>>> 0.17 315.97 0.57 135305 0.00 0.00
> >> >>>> Curl_num_addresses
> >> >>>> 0.17 316.54 0.57 181937907 0.00 0.00 addbyter
> >> >>>> 0.17 317.09 0.55 2374363 0.00 0.00
> multi_runsingle
> >> >>>> 0.16 317.62 0.53 541954 0.00 0.00
> >> >>>> ares__init_list_node
> >> >>>> 0.16 318.14 0.52 494412 0.00 0.00
> >> >>>> curl_multi_socket_action
> >> >>>> 0.14 318.60 0.46 33578897 0.00 0.00
> >> >>>> Curl_socket_check
> >> >>>> 0.13 319.04 0.44 102951052 0.00 0.00
> >> >>>> Curl_raw_toupper
> >> >>>> 0.13 319.48 0.44 787558 0.00 0.00
> Curl_readwrite
> >> >>>> 0.13 319.91 0.44 id_function
> >> >>>> 0.12 320.29 0.38 391390 0.00 0.00 Curl_hash_str
> >> >>>> 0.09 320.58 0.29 9216 0.00 0.00
> >> >>>> curl_multi_perform
> >> >>>> 0.08 320.86 0.28 2350908 0.00 0.00
> Curl_raw_equal
> >> >>>> 0.08 321.14 0.28 2019415 0.00 0.00 Curl_splay
> >> >>>> 0.08 321.40 0.26 520625 0.00 0.00
> >> >>>> ossl_connect_common
> >> >>>> 0.08 321.65 0.25 3532992 0.00 0.00
> Curl_pgrsUpdate
> >> >>>> 0.07 321.87 0.22 131778537 0.00 0.00
> curl_strequal
> >> >>>> 0.05 322.05 0.18 8954118 0.00 0.00 scan_response
> >> >>>> 0.05 322.23 0.18 533644 0.00 0.00
> >> >>>> ares_expand_name
> >> >>>> 0.05 322.40 0.17 2477345 0.00 0.00
> fd_key_compare
> >> >>>> 0.05 322.57 0.17 2429135 0.00 0.00 Curl_infof
> >> >>>> 0.05 322.74 0.17 130966 0.00 0.00
> singleipconnect
> >> >>>> 0.05 322.90 0.16 9292 0.00 0.00
> >> >>>> curl_multi_socket_all
> >> >>>> 0.05 323.06 0.16 270137 0.00 0.00
> >> >>>> ares__get_hostent
> >> >>>> 0.05 323.21 0.15 8943657 0.00 0.00 Curl_debug
> >> >>>> 0.04 323.35 0.14 262493 0.00 0.00
> >> >>>> Curl_ssl_getsessionid
> >> >>>> 0.04 323.48 0.13 566006 0.00 0.00
> socket_callback
> >> >>>> 0.04 323.61 0.13 33983864 0.00 0.00 curlx_tvdiff
> >> >>>> 0.04 323.74 0.13 132853 0.00 0.00
> >> >>>> Curl_http_readwrite_headers
> >> >>>> 0.04 323.86 0.12 344032 0.00 0.00 epoll_del
> >> >>>> 0.04 323.98 0.12 223477 0.00 0.00 Curl_poll
> >> >>>> 0.04 324.10 0.12 8526280 0.00 0.00
> Curl_raw_nequal
> >> >>>> 0.03 324.21 0.11 348839 0.00 0.00 event_del
> >> >>>>
> >> >>>>
> >> >>>>
> >> >>>>
> >> >>>> test 2:
> >> >>>>
> >> >>>> dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve
> >> >>>> *.com to one IP address
> >> >>>> number of domain names in all https URLs: 1
> >> >>>> number of client: 1000
> >> >>>> CPU usage: 75%
> >> >>>> SSL TPS: 1300
> >> >>>> SSL throughput: 700Mbps
> >> >>>>
> >> >>>> # gprof /usr/bin/curl-loader gmon.out -p | head -50
> >> >>>> Flat profile:
> >> >>>> Each sample counts as 0.01 seconds.
> >> >>>> % cumulative self self total
> >> >>>> time seconds seconds calls s/call s/call name
> >> >>>> 8.98 2.45 2.45 5127312 0.00 0.00
> dprintf_formatf
> >> >>>> 8.90 4.88 2.43 1292796503 0.00 0.00
> >> >>>> ares__is_list_empty
> >> >>>> 6.12 6.55 1.67 322461 0.00 0.00 create_conn
> >> >>>> 4.58 7.80 1.25 419872 0.00 0.00 ares_cancel
> >> >>>> 4.10 8.92 1.12 2100162 0.00 0.00
> Curl_readwrite
> >> >>>> 3.92 9.99 1.07 4527388 0.00 0.00
> Curl_hash_pick
> >> >>>> 3.43 10.93 0.94
> >> >>>> locking_function
> >> >>>> 3.04 11.76 0.83 909759 0.00 0.00
> >> >>>> curl_multi_socket_action
> >> >>>> 3.00 12.58 0.82 97252 0.00 0.00
> >> >>>> Curl_hash_clean_with_criterium
> >> >>>> 2.71 13.32 0.74 179704016 0.00 0.00
> >> >>>> Curl_raw_toupper
> >> >>>> 2.68 14.05 0.73 4646524 0.00 0.00
> multi_runsingle
> >> >>>> 2.44 14.71 0.67 13862992 0.00 0.00
> >> >>>> client_tracing_function
> >> >>>> 2.42 15.37 0.66 319735 0.00 0.00
> >> >>>> curl_multi_remove_handle
> >> >>>> 2.42 16.03 0.66 22 0.03 0.03
> >> >>>> ares__init_list_node
> >> >>>> 1.94 16.56 0.53 9930244 0.00 0.00
> >> >>>> hostcache_timestamp_remove
> >> >>>> 1.94 17.09 0.53 169320088 0.00 0.00 addbyter
> >> >>>> 1.80 17.58 0.49 id_function
> >> >>>> 1.36 17.95 0.37 7057146 0.00 0.00
> Curl_pgrsUpdate
> >> >>>> 1.36 18.32 0.37 16835 0.00 0.00
> >> >>>> curl_multi_socket_all
> >> >>>> 1.25 18.66 0.34 16719 0.00 0.00
> >> >>>> curl_multi_perform
> >> >>>> 1.21 18.99 0.33 329403 0.00 0.00
> >> >>>> Curl_http_readwrite_headers
> >> >>>> 1.17 19.31 0.32 4036624 0.00 0.00 Curl_splay
> >> >>>> 1.14 19.62 0.31 20838902 0.00 0.00
> Curl_raw_nequal
> >> >>>> 0.82 19.85 0.23 2764123 0.00 0.00 Curl_infof
> >> >>>> 0.81 20.07 0.22 5933857 0.00 0.00 ossl_recv
> >> >>>> 0.77 20.28 0.21 323112 0.00 0.00
> >> >>>> Curl_splayremovebyaddr
> >> >>>> 0.75 20.48 0.21 5943991 0.00 0.00 Curl_read
> >> >>>> 0.70 20.67 0.19 425174 0.00 0.00 event_del
> >> >>>> 0.70 20.86 0.19 98652 0.00 0.00
> >> >>>> Curl_if_is_interface_name
> >> >>>> 0.70 21.05 0.19 8677733 0.00 0.00
> >> >>>> Curl_socket_check
> >> >>>> 0.59 21.21 0.16 4129554 0.00 0.00
> fd_key_compare
> >> >>>> 0.57 21.37 0.16 14017019 0.00 0.00 Curl_debug
> >> >>>> 0.55 21.52 0.15 9376861 0.00 0.00
> >> >>>> stat_data_in_add
> >> >>>> 0.55 21.67 0.15 6840468 0.00 0.00 Curl_timeleft
> >> >>>> 0.55 21.82 0.15 2054602 0.00 0.00
> Curl_raw_equal
> >> >>>> 0.55 21.97 0.15 1291611 0.00 0.00 Curl_expire
> >> >>>> 0.51 22.11 0.14 16570063 0.00 0.00 curlx_tvnow
> >> >>>> 0.51 22.25 0.14 13863284 0.00 0.00 scan_response
> >> >>>> 0.51 22.39 0.14 5498864 0.00 0.00 Curl_setopt
> >> >>>> 0.48 22.52 0.13 97490291 0.00 0.00 curl_strequal
> >> >>>> 0.44 22.64 0.12 324253 0.00 0.00 Curl_http
> >> >>>> 0.44 22.76 0.12 394386 0.00 0.00
> >> >>>> ossl_connect_common
> >> >>>> 0.40 22.87 0.11 3581335 0.00 0.00 Curl_getinfo
> >> >>>> 0.38 22.97 0.11 26983092 0.00 0.00
> alloc_addbyter
> >> >>>> 0.37 23.07 0.10 8729044 0.00 0.00
> >> >>>> Curl_client_write
> >> >>>>
> >> >>>>
> >> >>>>
> >> >>>>
> ------------------------------------------------------------------------------
> >> >>>> Try before you buy = See our experts in action!
> >> >>>> The most comprehensive online learning library for Microsoft
> >> >>>> developers
> >> >>>> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3,
> >> >>>> MVC3,
> >> >>>> Metro Style Apps, more. Free future releases when you subscribe
> now!
> >> >>>> http://p.sf.net/sfu/learndevnow-dev2
> >> >>>> _______________________________________________
> >> >>>> curl-loader-devel mailing list
> >> >>>> cur...@li...
> >> >>>> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel
> >> >>>>
> >> >>>
> >> >>>
> >> >>>
> >> >>> --
> >> >>> Regards,
> >> >>> Robert Iakobashvili, Ph.D.
> >> >>>
> >> >>> Home: http://www.ghotit.com
> >> >>> Blog: http://www.ghotit.com/category/ghotit-blogs/
> >> >>> Twitter: http://twitter.com/ghotit
> >> >>> Facebook: http://facebook.com/ghotit
> >> >>>
> ......................................................................
> >> >>> Ghotit Dyslexia
> >> >>> Assistive technology that understands you
> >> >>>
> ......................................................................
> >> >>>
> >> >>>
> >> >>>
> >> >>>
> ------------------------------------------------------------------------------
> >> >>> Try before you buy = See our experts in action!
> >> >>> The most comprehensive online learning library for Microsoft
> >> >>> developers
> >> >>> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3,
> >> >>> MVC3,
> >> >>> Metro Style Apps, more. Free future releases when you subscribe now!
> >> >>> http://p.sf.net/sfu/learndevnow-dev2
> >> >>> _______________________________________________
> >> >>> curl-loader-devel mailing list
> >> >>> cur...@li...
> >> >>> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel
> >> >>>
> >> >>
> >> >>
> >> >>
> >> >>
> ------------------------------------------------------------------------------
> >> >> Try before you buy = See our experts in action!
> >> >> The most comprehensive online learning library for Microsoft
> developers
> >> >> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3,
> >> >> MVC3,
> >> >> Metro Style Apps, more. Free future releases when you subscribe now!
> >> >> http://p.sf.net/sfu/learndevnow-dev2
> >> >> _______________________________________________
> >> >> curl-loader-devel mailing list
> >> >> cur...@li...
> >> >> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel
> >> >>
> >> >
> >> >
> >> >
> >> > --
> >> > Regards,
> >> > Robert Iakobashvili, Ph.D.
> >> >
> >> > Home: http://www.ghotit.com
> >> > Blog: http://www.ghotit.com/category/ghotit-blogs/
> >> > Twitter: http://twitter.com/ghotit
> >> > Facebook: http://facebook.com/ghotit
> >> > ......................................................................
> >> > Ghotit Dyslexia
> >> > Assistive technology that understands you
> >> > ......................................................................
> >>
> >>
> >>
> >> --
> >> Regards,
> >> Robert Iakobashvili, Ph.D.
> >>
> >> Home: http://www.ghotit.com
> >> Blog: http://www.ghotit.com/category/ghotit-blogs/
> >> Twitter: http://twitter.com/ghotit
> >> Facebook: http://facebook.com/ghotit
> >> ......................................................................
> >> Ghotit Dyslexia
> >> Assistive technology that understands you
> >> ......................................................................
> >>
> >>
> >>
> ------------------------------------------------------------------------------
> >> Try before you buy = See our experts in action!
> >> The most comprehensive online learning library for Microsoft developers
> >> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3,
> >> Metro Style Apps, more. Free future releases when you subscribe now!
> >> http://p.sf.net/sfu/learndevnow-dev2
> >> _______________________________________________
> >> curl-loader-devel mailing list
> >> cur...@li...
> >> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel
> >
> >
> >
> >
> ------------------------------------------------------------------------------
> > Keep Your Developer Skills Current with LearnDevNow!
> > The most comprehensive online learning library for Microsoft developers
> > is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3,
> > Metro Style Apps, more. Free future releases when you subscribe now!
> > http://p.sf.net/sfu/learndevnow-d2d
> > _______________________________________________
> > curl-loader-devel mailing list
> > cur...@li...
> > https://lists.sourceforge.net/lists/listinfo/curl-loader-devel
> >
>
>
>
> --
> Regards,
> Robert Iakobashvili, Ph.D.
>
> Home: http://www.ghotit.com
> Blog: http://www.ghotit.com/category/ghotit-blogs/
> Twitter: http://twitter.com/ghotit
> Facebook: http://facebook.com/ghotit
> ......................................................................
> Ghotit Dyslexia
> Assistive technology that understands you
> ......................................................................
>
>
> ------------------------------------------------------------------------------
> Keep Your Developer Skills Current with LearnDevNow!
> The most comprehensive online learning library for Microsoft developers
> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3,
> Metro Style Apps, more. Free future releases when you subscribe now!
> http://p.sf.net/sfu/learndevnow-d2d
> _______________________________________________
> curl-loader-devel mailing list
> cur...@li...
> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel
>
|
|
From: Robert I. <cor...@gm...> - 2012-03-06 05:28:39
|
Several questions:
1. Could you please run top and see if there are other processes burning CPU?
2. What is the number of loading clients that you use?
3. Could you try curl-loader-0.52 if it compiles for you?
If not, please try to place curl-loader-7.19.7.tar.gz and c-ares-16.0.tar.gz,
update their versions in Makefile and rebuild by make cleanall; make
4. Is it correct to say that for HTTP URLs (not HTTPS) you are getting
a good throughput
and CAPS with curl-loader-0.56?
Thanks,
Robert
On Tue, Mar 6, 2012 at 3:52 AM, Fred Huang <di...@gm...> wrote:
> TIMEOUT -1 -> 60 works a little better but no performance improvement on
> CAPS or throughput.
>
> test 5:
> enable c-ares;
> curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, 60);
>
> dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve *.com to
> one IP address
> number of domain names in URLs: 780,000
> number of client: 1000
> CPU usage: 74%
> cpu% irq% sirq% sys% iowt% mem_used buf&cached
> 74.0 0.0 0.7 4.2 0.0 3530.6Mb 250.8Mb
> SSL TPS: 360
> SSL throughput: 220Mbps
>
> # gprof /usr/bin/curl-loader gmon.out -p | head -50
> Flat profile:
> Each sample counts as 0.01 seconds.
> % cumulative self self total
> time seconds seconds calls s/call s/call name
> 26.63 23.13 23.13 119226 0.00 0.00
> Curl_hash_clean_with_criterium
> 20.34 40.80 17.67 317792146 0.00 0.00
> hostcache_timestamp_remove
> 11.27 50.59 9.79 2579903 0.00 0.00 Curl_hash_pick
> 7.84 57.40 6.81 120606 0.00 0.00 create_conn
> 5.65 62.31 4.91 119117401 0.00 0.00 Curl_str_key_compare
> 4.98 66.64 4.33 329829 0.00 0.00 Curl_hash_add
> 2.80 69.08 2.44 4476606 0.00 0.00 dprintf_formatf
> 2.57 71.31 2.23 124019 0.00 0.00 ConnectionStore
> 1.50 72.61 1.30 740997609 0.00 0.00 ares__is_list_empty
> 1.16 73.61 1.01 locking_function
> 0.76 74.27 0.66 238744 0.00 0.00 ares_cancel
> 0.70 74.88 0.61 457564 0.00 0.00
> curl_multi_socket_action
> 0.67 75.46 0.58 118616 0.00 0.00
> curl_multi_remove_handle
> 0.58 75.97 0.51 8211502 0.00 0.00
> client_tracing_function
> 0.53 76.43 0.46 733561 0.00 0.00 Curl_readwrite
> 0.51 76.87 0.44 2183179 0.00 0.00 multi_runsingle
> 0.51 77.31 0.44 166585567 0.00 0.00 addbyter
> 0.44 77.69 0.38 2175805 0.00 0.00 Curl_raw_equal
> 0.43 78.06 0.38 id_function
> 0.40 78.41 0.35 29789346 0.00 0.00 Curl_socket_check
> 0.40 78.76 0.35 479671 0.00 0.00 ossl_connect_common
> 0.40 79.11 0.35 490639 0.00 0.00 ares__init_list_node
> 0.33 79.40 0.29 94197670 0.00 0.00 Curl_raw_toupper
> 0.33 79.69 0.29 3268031 0.00 0.00 Curl_pgrsUpdate
> 0.28 79.93 0.24 8797 0.00 0.00 curl_multi_perform
> 0.27 80.17 0.24 2233641 0.00 0.00 Curl_infof
> 0.26 80.40 0.23 1839837 0.00 0.00 Curl_splay
> 0.24 80.61 0.21 121713028 0.00 0.00 curl_strequal
> 0.21 80.79 0.18 243333 0.00 0.00 Curl_ssl_getsessionid
> 0.18 80.95 0.16 122180 0.00 0.00 Curl_num_addresses
> 0.17 81.09 0.15 8271666 0.00 0.00 Curl_debug
> 0.16 81.23 0.14 18902452 0.00 0.00 alloc_addbyter
> 0.16 81.37 0.14 8851 0.00 0.00 curl_multi_socket_all
> 0.14 81.50 0.13 2236769 0.00 0.00 Curl_read
> 0.14 81.62 0.12 2271700 0.00 0.00 fd_key_compare
> 0.14 81.74 0.12 323167 0.00 0.00 event_del
> 0.14 81.86 0.12 205436 0.00 0.00 read_udp_packets
> 0.14 81.98 0.12 200979 0.00 0.00 processfds
> 0.13 82.09 0.11 123886 0.00 0.00
> Curl_if_is_interface_name
> 0.13 82.20 0.11 120350 0.00 0.00 Curl_http
> 0.13 82.31 0.11 ssl_tls_trace
> 0.12 82.41 0.10 119229 0.00 0.00 Curl_splayremovebyaddr
> 0.12 82.51 0.10 7798717 0.00 0.00 Curl_raw_nequal
> 0.12 82.61 0.10 784485 0.00 0.00 event_add
> 0.10 82.70 0.09 643482 0.00 0.00 Curl_expire
>
>
>
>
>
> test 6:
> enable c-ares;
> curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, 60);
> CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.23.0
>
> dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve *.com to
> one IP address
> number of domain names in URLs: 780,000
> number of client: 1000
> CPU usage: 86%
> cpu% irq% sirq% sys% iowt% mem_used buf&cached
> 85.9 0.0 0.2 2.1 0.0 3537.2Mb 261.4Mb
> SSL TPS: 360
> SSL throughput: 200Mbps
>
> # gprof /usr/bin/curl-loader gmon.out -p | head -20
> Flat profile:
> Each sample counts as 0.01 seconds.
> % cumulative self self total
> time seconds seconds calls s/call s/call name
> 27.19 19.96 19.96 103472 0.00 0.00
> Curl_hash_clean_with_criterium
> 21.73 35.91 15.95 274168981 0.00 0.00
> hostcache_timestamp_remove
> 10.33 43.49 7.58 2256441 0.00 0.00 Curl_hash_pick
> 7.21 48.78 5.29 104540 0.00 0.00 create_conn
> 6.20 53.33 4.55 103413405 0.00 0.00 Curl_str_key_compare
> 4.88 56.91 3.58 287109 0.00 0.00 Curl_hash_add
> 3.02 59.13 2.22 3797855 0.00 0.00 dprintf_formatf
> 2.10 60.67 1.54 107711 0.00 0.00 ConnectionStore
> 1.65 61.88 1.21 642725327 0.00 0.00 ares__is_list_empty
> 1.19 62.76 0.88 locking_function
> 0.87 63.40 0.64 207817 0.00 0.00 ares_cancel
> 0.56 63.81 0.41 404108 0.00 0.00
> curl_multi_socket_action
> 0.54 64.21 0.40 425603 0.00 0.00 ares__init_list_node
> 0.53 64.60 0.39 7068112 0.00 0.00
> client_tracing_function
> 0.49 64.96 0.36 1908587 0.00 0.00 multi_runsingle
>
>
>
> test 7:
> enable c-ares;
> curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, -1);
> CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.23.0
>
> dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve *.com to
> one IP address
> number of domain names in URLs: 780,000
> number of client: 1000
> CPU usage: 86%
> cpu% irq% sirq% sys% iowt% mem_used buf&cached
> 86.6 0.0 0.6 3.3 0.0 3624.8Mb 263.0Mb
> SSL TPS: 300
> SSL throughput: 200Mbps
>
> # gprof /usr/bin/curl-loader gmon.out -p | head -20
> Flat profile:
> Each sample counts as 0.01 seconds.
> % cumulative self self total
> time seconds seconds calls s/call s/call name
> 39.83 369.71 369.71 207173 0.00 0.00
> Curl_hash_clean_with_criterium
> 24.27 594.98 225.27 3198337733 0.00 0.00
> hostcache_timestamp_remove
> 13.75 722.59 127.61 4312864 0.00 0.00 Curl_hash_pick
> 9.19 807.87 85.28 546507 0.00 0.00 Curl_hash_add
> 6.14 864.88 57.01 1022419684 0.00 0.00 Curl_str_key_compare
> 2.06 884.00 19.12 203749 0.00 0.00 create_conn
> 0.54 888.99 4.99 213306 0.00 0.00 ConnectionStore
> 0.49 893.50 4.51 7331251 0.00 0.00 dprintf_formatf
> 0.29 896.16 2.66 1287182329 0.00 0.00 ares__is_list_empty
> 0.23 898.28 2.12 locking_function
> 0.20 900.10 1.82 212210 0.00 0.00 Curl_num_addresses
> 0.17 901.66 1.56 198385 0.00 0.00
> curl_multi_remove_handle
> 0.12 902.81 1.15 411114 0.00 0.00 ares_cancel
> 0.12 903.95 1.14 3683448 0.00 0.00 multi_runsingle
> 0.11 904.97 1.03 13664120 0.00 0.00
> client_tracing_function
>
>
> test 8:
> enable c-ares;
> curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, 60);
> CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.24.0
>
> dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve *.com to
> one IP address
> number of domain names in URLs: 780,000
> number of client: 1000
> CPU usage: 75%
> cpu% irq% sirq% sys% iowt% mem_used buf&cached
> 75.2 0.0 0.2 2.1 0.0 3537.2Mb 261.4Mb
> SSL TPS: 380
> SSL throughput: 220Mbps
>
> # gprof /usr/bin/curl-loader gmon.out -p | head -20
> Flat profile:
> Each sample counts as 0.01 seconds.
> % cumulative self self total
> time seconds seconds calls s/call s/call name
> 27.58 12.31 12.31 71140 0.00 0.00
> Curl_hash_clean_with_criterium
> 19.09 20.83 8.52 176729692 0.00 0.00
> hostcache_timestamp_remove
> 10.44 25.49 4.66 1589309 0.00 0.00 Curl_hash_pick
> 6.85 28.55 3.06 73204 0.00 0.00 create_conn
> 5.56 31.03 2.48 67620530 0.00 0.00 Curl_str_key_compare
> 4.41 33.00 1.97 200684 0.00 0.00 Curl_hash_add
> 3.56 34.59 1.59 2717956 0.00 0.00 dprintf_formatf
> 1.72 35.36 0.77 75247 0.00 0.00 ConnectionStore
> 1.70 36.12 0.76 444342256 0.00 0.00 ares__is_list_empty
> 1.23 36.67 0.55 locking_function
> 1.16 37.19 0.52 143342 0.00 0.00 ares_cancel
> 0.99 37.63 0.44 72211 0.00 0.00
> curl_multi_remove_handle
> 0.74 37.96 0.33 1333371 0.00 0.00 multi_runsingle
> 0.68 38.27 0.31 101147798 0.00 0.00 addbyter
> 0.58 38.53 0.26 288737 0.00 0.00
> curl_multi_socket_action
>
>
>
> test 9:
> enable c-ares;
> curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, -1);
> CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.24.0
>
> dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve *.com to
> one IP address
> number of domain names in URLs: 780,000
> number of client: 1000
> CPU usage: 75%
> cpu% irq% sirq% sys% iowt% mem_used buf&cached
> 75.1 0.0 0.6 4.1 0.0 3545.2Mb 260.1Mb
> SSL TPS: 380
> SSL throughput: 200Mbps
>
> # gprof /usr/bin/curl-loader gmon.out -p | head -20
> Flat profile:
> Each sample counts as 0.01 seconds.
> % cumulative self self total
> time seconds seconds calls s/call s/call name
> 34.00 31.13 31.13 72495 0.00 0.00
> Curl_hash_clean_with_criterium
> 23.64 52.78 21.65 386070412 0.00 0.00
> hostcache_timestamp_remove
> 13.28 64.94 12.16 1619440 0.00 0.00 Curl_hash_pick
> 5.92 70.36 5.42 137653129 0.00 0.00 Curl_str_key_compare
> 5.50 75.40 5.04 203018 0.00 0.00 Curl_hash_add
> 4.55 79.57 4.17 74618 0.00 0.00 create_conn
> 1.70 81.13 1.56 2757148 0.00 0.00 dprintf_formatf
> 0.98 82.03 0.90 76945 0.00 0.00 ConnectionStore
> 0.95 82.90 0.87 455775100 0.00 0.00 ares__is_list_empty
> 0.57 83.42 0.53 locking_function
> 0.39 83.78 0.36 146742 0.00 0.00 ares_cancel
> 0.35 84.10 0.32 102968806 0.00 0.00 addbyter
> 0.35 84.42 0.32 456587 0.00 0.00 Curl_readwrite
> 0.35 84.74 0.32 73133 0.00 0.00
> curl_multi_remove_handle
> 0.34 85.05 0.31 1360545 0.00 0.00 multi_runsingle
>
>
> 2012/3/6 Robert Iakobashvili <cor...@gm...>
>>
>> If it will not be helpful this means that recent
>> changes in libcurl library have broken DNS interface.
>>
>> You can try a previos version and the latest libcurl, like:
>>
>> http://curl.haxx.se/download/curl-7.23.0.tar.gz
>> and
>> http://curl.haxx.se/downloadcurl-7.24.0.tar.gz
>>
>> by:
>>
>> 1. downloading a libcurl version and placing it to
>> curl-loader-0.56/packages ;
>> 2. correcting the Makefile in curl-loader-0.56
>> CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.24 or CURL_VER:=7.23
>> 3. make cleanall; make
>>
>> Please, send us your observations.
>> Thanks!
>> Robert
>>
>> On Mon, Mar 5, 2012 at 6:57 PM, Robert Iakobashvili <cor...@gm...>
>> wrote:
>> > If so, please try to change
>> >
>> > loader.c, line 578 which looks like:
>> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, -1);
>> >
>> > 1. instead of -1 try to set some value like 60
>> > 2. make clean; make
>> >
>> > Thanks,
>> > Robert
>> >
>> > On Mon, Mar 5, 2012 at 6:19 PM, Fred Huang <di...@gm...> wrote:
>> >>
>> >> Looks like the bottleneck comes from libcurl other than c-ares...
>> >> test 3 (no ares patch applied):
>> >> dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve *.com
>> >> to one IP address
>> >> number of domain names in URLs: 780,000
>> >> number of client: 1000
>> >> CPU usage: 85%
>> >> cpu% irq% sirq% sys% iowt% mem_used buf&cached
>> >> 85.6 0.0 0.8 5.9 0.0 3334.3Mb 112.0Mb
>> >> SSL TPS: 320
>> >> SSL throughput: 200Mbps
>> >>
>> >> # gprof /usr/bin/curl-loader gmon.out -p | head -50
>> >> Flat profile:
>> >> Each sample counts as 0.01 seconds.
>> >> % cumulative self self total
>> >> time seconds seconds calls s/call s/call name
>> >> 40.80 120.16 120.16 116079 0.00 0.00
>> >> Curl_hash_clean_with_criterium
>> >> 24.46 192.18 72.02 983166323 0.00 0.00
>> >> hostcache_timestamp_remove
>> >> 13.30 231.35 39.17 2059341 0.00 0.00 Curl_hash_pick
>> >> 6.45 250.35 19.00 240087 0.00 0.00 Curl_hash_add
>> >> 4.81 264.51 14.16 334257102 0.00 0.00
>> >> Curl_str_key_compare
>> >> 3.40 274.52 10.01 119514 0.00 0.00 create_conn
>> >> 0.93 277.25 2.73 4478910 0.00 0.00 dprintf_formatf
>> >> 0.90 279.91 2.66 122209 0.00 0.00 ConnectionStore
>> >> 0.40 281.09 1.18 locking_function
>> >> 0.27 281.88 0.79 114759 0.00 0.00
>> >> curl_multi_remove_handle
>> >> 0.21 282.51 0.63 7993664 0.00 0.00
>> >> client_tracing_function
>> >> 0.21 283.12 0.61 122685 0.00 0.00
>> >> Curl_num_addresses
>> >> 0.19 283.68 0.56 397028 0.00 0.00
>> >> curl_multi_socket_action
>> >> 0.19 284.23 0.55 2036116 0.00 0.00 multi_runsingle
>> >> 0.17 284.72 0.49 703687 0.00 0.00 Curl_readwrite
>> >> 0.16 285.19 0.47 id_function
>> >> 0.15 285.62 0.43 29692029 0.00 0.00 Curl_socket_check
>> >> 0.14 286.02 0.40 91973856 0.00 0.00 Curl_raw_toupper
>> >> 0.13 286.40 0.38 164664144 0.00 0.00 addbyter
>> >> 0.13 286.77 0.37 2180797 0.00 0.00 Curl_infof
>> >> 0.11 287.10 0.33 122675 0.00 0.00
>> >> Curl_if_is_interface_name
>> >> 0.10 287.39 0.29 471318 0.00 0.00
>> >> ossl_connect_common
>> >> 0.10 287.67 0.28 2113528 0.00 0.00 Curl_raw_equal
>> >> 0.09 287.93 0.26 8676 0.00 0.02
>> >> curl_multi_socket_all
>> >> 0.08 288.17 0.24 1663262 0.00 0.00 Curl_splay
>> >> 0.08 288.41 0.24 8505 0.00 0.01
>> >> curl_multi_perform
>> >> 0.07 288.63 0.22 3114576 0.00 0.00 Curl_pgrsUpdate
>> >> 0.06 288.80 0.17 119637 0.00 0.00 singleipconnect
>> >> 0.06 288.97 0.17 7994818 0.00 0.00 Curl_debug
>> >> 0.05 289.13 0.16 235820 0.00 0.00 event_del
>> >> 0.05 289.29 0.16 119393071 0.00 0.00 curl_strequal
>> >> 0.05 289.45 0.16 18535473 0.00 0.00 alloc_addbyter
>> >> 0.05 289.61 0.16 356074 0.00 0.00 Curl_hash_str
>> >> 0.05 289.76 0.15 1797233 0.00 0.00 fd_key_compare
>> >> 0.05 289.91 0.15 234966 0.00 0.00
>> >> Curl_ssl_getsessionid
>> >> 0.04 290.04 0.13 585853 0.00 0.00
>> >> Curl_llist_destroy
>> >> 0.04 290.16 0.12 240826 0.00 0.00 Curl_resolv
>> >> 0.03 290.26 0.10 7514328 0.00 0.00 Curl_raw_nequal
>> >> 0.03 290.36 0.10 2214436 0.00 0.00 Curl_read
>> >> 0.03 290.46 0.10 116872 0.00 0.00
>> >> Curl_http_readwrite_headers
>> >> 0.03 290.56 0.10 ssl_tls_trace
>> >> 0.03 290.66 0.10 8013097 0.00 0.00 scan_response
>> >> 0.03 290.75 0.09 29488555 0.00 0.00 curlx_tvdiff
>> >> 0.03 290.84 0.09 2142468 0.00 0.00 Curl_setopt
>> >> 0.03 290.93 0.09 476299 0.00 0.00 Curl_expire
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> 2012/3/5 Robert Iakobashvili <cor...@gm...>
>> >>>
>> >>> Dear Fred Huang,
>> >>>
>> >>> Thanks for your posting.
>> >>>
>> >>> Could you please check if this is due to some badness at cares?
>> >>> Thanks.
>> >>>
>> >>> Let's exclude usage of cares based DNS resolving by applying the patch
>> >>> attached
>> >>> and use instead global DNS cache.
>> >>>
>> >>> cd ./curl-loader-0.56
>> >>> patch -p1 < path-to/disable-cares-use-global-dns.patch
>> >>> make cleanall; make
>> >>>
>> >>> See if it helps.
>> >>>
>> >>> Global cache, unfortunately, is not thread safe.
>> >>>
>> >>> If it helps, we'll try to restore the previous version of cares.
>> >>>
>> >>> Thanks,
>> >>> Robert
>> >>>
>> >>> On Mon, Mar 5, 2012 at 4:28 PM, Fred Huang <di...@gm...> wrote:
>> >>>>
>> >>>> test 1:
>> >>>>
>> >>>> dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve
>> >>>> *.com to one IP address
>> >>>> number of domain names in all https URLs: 780,000
>> >>>> number of client: 1000
>> >>>> CPU usage: 78%
>> >>>> cpu% irq% sirq% sys% iowt% mem_used buf&cached
>> >>>> 78.8 0.0 0.6 3.8 0.0 3640.9Mb 282.1Mb
>> >>>> SSL TPS: 300
>> >>>> SSL throughput: 200Mbps
>> >>>>
>> >>>> # gprof /usr/bin/curl-loader gmon.out -p | head -50
>> >>>> Flat profile:
>> >>>> Each sample counts as 0.01 seconds.
>> >>>> % cumulative self self total
>> >>>> time seconds seconds calls s/call s/call name
>> >>>> 38.55 127.10 127.10 130975 0.00 0.00
>> >>>> Curl_hash_clean_with_criterium
>> >>>> 24.63 208.32 81.22 1247528407 0.00 0.00
>> >>>> hostcache_timestamp_remove
>> >>>> 13.08 251.44 43.12 2791988 0.00 0.00 Curl_hash_pick
>> >>>> 7.30 275.51 24.07 352490 0.00 0.00 Curl_hash_add
>> >>>> 6.01 295.31 19.80 410760770 0.00 0.00
>> >>>> Curl_str_key_compare
>> >>>> 2.95 305.04 9.73 131015 0.00 0.00 create_conn
>> >>>> 0.88 307.94 2.90 4841390 0.00 0.00 dprintf_formatf
>> >>>> 0.74 310.38 2.44 136036 0.00 0.00 ConnectionStore
>> >>>> 0.48 311.96 1.58 814604911 0.00 0.00
>> >>>> ares__is_list_empty
>> >>>> 0.36 313.16 1.20
>> >>>> locking_function
>> >>>> 0.27 314.06 0.90 261608 0.00 0.00 ares_cancel
>> >>>> 0.22 314.79 0.73 127417 0.00 0.00
>> >>>> curl_multi_remove_handle
>> >>>> 0.19 315.40 0.62 8906301 0.00 0.00
>> >>>> client_tracing_function
>> >>>> 0.17 315.97 0.57 135305 0.00 0.00
>> >>>> Curl_num_addresses
>> >>>> 0.17 316.54 0.57 181937907 0.00 0.00 addbyter
>> >>>> 0.17 317.09 0.55 2374363 0.00 0.00 multi_runsingle
>> >>>> 0.16 317.62 0.53 541954 0.00 0.00
>> >>>> ares__init_list_node
>> >>>> 0.16 318.14 0.52 494412 0.00 0.00
>> >>>> curl_multi_socket_action
>> >>>> 0.14 318.60 0.46 33578897 0.00 0.00
>> >>>> Curl_socket_check
>> >>>> 0.13 319.04 0.44 102951052 0.00 0.00
>> >>>> Curl_raw_toupper
>> >>>> 0.13 319.48 0.44 787558 0.00 0.00 Curl_readwrite
>> >>>> 0.13 319.91 0.44 id_function
>> >>>> 0.12 320.29 0.38 391390 0.00 0.00 Curl_hash_str
>> >>>> 0.09 320.58 0.29 9216 0.00 0.00
>> >>>> curl_multi_perform
>> >>>> 0.08 320.86 0.28 2350908 0.00 0.00 Curl_raw_equal
>> >>>> 0.08 321.14 0.28 2019415 0.00 0.00 Curl_splay
>> >>>> 0.08 321.40 0.26 520625 0.00 0.00
>> >>>> ossl_connect_common
>> >>>> 0.08 321.65 0.25 3532992 0.00 0.00 Curl_pgrsUpdate
>> >>>> 0.07 321.87 0.22 131778537 0.00 0.00 curl_strequal
>> >>>> 0.05 322.05 0.18 8954118 0.00 0.00 scan_response
>> >>>> 0.05 322.23 0.18 533644 0.00 0.00
>> >>>> ares_expand_name
>> >>>> 0.05 322.40 0.17 2477345 0.00 0.00 fd_key_compare
>> >>>> 0.05 322.57 0.17 2429135 0.00 0.00 Curl_infof
>> >>>> 0.05 322.74 0.17 130966 0.00 0.00 singleipconnect
>> >>>> 0.05 322.90 0.16 9292 0.00 0.00
>> >>>> curl_multi_socket_all
>> >>>> 0.05 323.06 0.16 270137 0.00 0.00
>> >>>> ares__get_hostent
>> >>>> 0.05 323.21 0.15 8943657 0.00 0.00 Curl_debug
>> >>>> 0.04 323.35 0.14 262493 0.00 0.00
>> >>>> Curl_ssl_getsessionid
>> >>>> 0.04 323.48 0.13 566006 0.00 0.00 socket_callback
>> >>>> 0.04 323.61 0.13 33983864 0.00 0.00 curlx_tvdiff
>> >>>> 0.04 323.74 0.13 132853 0.00 0.00
>> >>>> Curl_http_readwrite_headers
>> >>>> 0.04 323.86 0.12 344032 0.00 0.00 epoll_del
>> >>>> 0.04 323.98 0.12 223477 0.00 0.00 Curl_poll
>> >>>> 0.04 324.10 0.12 8526280 0.00 0.00 Curl_raw_nequal
>> >>>> 0.03 324.21 0.11 348839 0.00 0.00 event_del
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>> test 2:
>> >>>>
>> >>>> dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve
>> >>>> *.com to one IP address
>> >>>> number of domain names in all https URLs: 1
>> >>>> number of client: 1000
>> >>>> CPU usage: 75%
>> >>>> SSL TPS: 1300
>> >>>> SSL throughput: 700Mbps
>> >>>>
>> >>>> # gprof /usr/bin/curl-loader gmon.out -p | head -50
>> >>>> Flat profile:
>> >>>> Each sample counts as 0.01 seconds.
>> >>>> % cumulative self self total
>> >>>> time seconds seconds calls s/call s/call name
>> >>>> 8.98 2.45 2.45 5127312 0.00 0.00 dprintf_formatf
>> >>>> 8.90 4.88 2.43 1292796503 0.00 0.00
>> >>>> ares__is_list_empty
>> >>>> 6.12 6.55 1.67 322461 0.00 0.00 create_conn
>> >>>> 4.58 7.80 1.25 419872 0.00 0.00 ares_cancel
>> >>>> 4.10 8.92 1.12 2100162 0.00 0.00 Curl_readwrite
>> >>>> 3.92 9.99 1.07 4527388 0.00 0.00 Curl_hash_pick
>> >>>> 3.43 10.93 0.94
>> >>>> locking_function
>> >>>> 3.04 11.76 0.83 909759 0.00 0.00
>> >>>> curl_multi_socket_action
>> >>>> 3.00 12.58 0.82 97252 0.00 0.00
>> >>>> Curl_hash_clean_with_criterium
>> >>>> 2.71 13.32 0.74 179704016 0.00 0.00
>> >>>> Curl_raw_toupper
>> >>>> 2.68 14.05 0.73 4646524 0.00 0.00 multi_runsingle
>> >>>> 2.44 14.71 0.67 13862992 0.00 0.00
>> >>>> client_tracing_function
>> >>>> 2.42 15.37 0.66 319735 0.00 0.00
>> >>>> curl_multi_remove_handle
>> >>>> 2.42 16.03 0.66 22 0.03 0.03
>> >>>> ares__init_list_node
>> >>>> 1.94 16.56 0.53 9930244 0.00 0.00
>> >>>> hostcache_timestamp_remove
>> >>>> 1.94 17.09 0.53 169320088 0.00 0.00 addbyter
>> >>>> 1.80 17.58 0.49 id_function
>> >>>> 1.36 17.95 0.37 7057146 0.00 0.00 Curl_pgrsUpdate
>> >>>> 1.36 18.32 0.37 16835 0.00 0.00
>> >>>> curl_multi_socket_all
>> >>>> 1.25 18.66 0.34 16719 0.00 0.00
>> >>>> curl_multi_perform
>> >>>> 1.21 18.99 0.33 329403 0.00 0.00
>> >>>> Curl_http_readwrite_headers
>> >>>> 1.17 19.31 0.32 4036624 0.00 0.00 Curl_splay
>> >>>> 1.14 19.62 0.31 20838902 0.00 0.00 Curl_raw_nequal
>> >>>> 0.82 19.85 0.23 2764123 0.00 0.00 Curl_infof
>> >>>> 0.81 20.07 0.22 5933857 0.00 0.00 ossl_recv
>> >>>> 0.77 20.28 0.21 323112 0.00 0.00
>> >>>> Curl_splayremovebyaddr
>> >>>> 0.75 20.48 0.21 5943991 0.00 0.00 Curl_read
>> >>>> 0.70 20.67 0.19 425174 0.00 0.00 event_del
>> >>>> 0.70 20.86 0.19 98652 0.00 0.00
>> >>>> Curl_if_is_interface_name
>> >>>> 0.70 21.05 0.19 8677733 0.00 0.00
>> >>>> Curl_socket_check
>> >>>> 0.59 21.21 0.16 4129554 0.00 0.00 fd_key_compare
>> >>>> 0.57 21.37 0.16 14017019 0.00 0.00 Curl_debug
>> >>>> 0.55 21.52 0.15 9376861 0.00 0.00
>> >>>> stat_data_in_add
>> >>>> 0.55 21.67 0.15 6840468 0.00 0.00 Curl_timeleft
>> >>>> 0.55 21.82 0.15 2054602 0.00 0.00 Curl_raw_equal
>> >>>> 0.55 21.97 0.15 1291611 0.00 0.00 Curl_expire
>> >>>> 0.51 22.11 0.14 16570063 0.00 0.00 curlx_tvnow
>> >>>> 0.51 22.25 0.14 13863284 0.00 0.00 scan_response
>> >>>> 0.51 22.39 0.14 5498864 0.00 0.00 Curl_setopt
>> >>>> 0.48 22.52 0.13 97490291 0.00 0.00 curl_strequal
>> >>>> 0.44 22.64 0.12 324253 0.00 0.00 Curl_http
>> >>>> 0.44 22.76 0.12 394386 0.00 0.00
>> >>>> ossl_connect_common
>> >>>> 0.40 22.87 0.11 3581335 0.00 0.00 Curl_getinfo
>> >>>> 0.38 22.97 0.11 26983092 0.00 0.00 alloc_addbyter
>> >>>> 0.37 23.07 0.10 8729044 0.00 0.00
>> >>>> Curl_client_write
>> >>>>
>> >>>>
>> >>>>
>> >>>> ------------------------------------------------------------------------------
>> >>>> Try before you buy = See our experts in action!
>> >>>> The most comprehensive online learning library for Microsoft
>> >>>> developers
>> >>>> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3,
>> >>>> MVC3,
>> >>>> Metro Style Apps, more. Free future releases when you subscribe now!
>> >>>> http://p.sf.net/sfu/learndevnow-dev2
>> >>>> _______________________________________________
>> >>>> curl-loader-devel mailing list
>> >>>> cur...@li...
>> >>>> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel
>> >>>>
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> Regards,
>> >>> Robert Iakobashvili, Ph.D.
>> >>>
>> >>> Home: http://www.ghotit.com
>> >>> Blog: http://www.ghotit.com/category/ghotit-blogs/
>> >>> Twitter: http://twitter.com/ghotit
>> >>> Facebook: http://facebook.com/ghotit
>> >>> ......................................................................
>> >>> Ghotit Dyslexia
>> >>> Assistive technology that understands you
>> >>> ......................................................................
>> >>>
>> >>>
>> >>>
>> >>> ------------------------------------------------------------------------------
>> >>> Try before you buy = See our experts in action!
>> >>> The most comprehensive online learning library for Microsoft
>> >>> developers
>> >>> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3,
>> >>> MVC3,
>> >>> Metro Style Apps, more. Free future releases when you subscribe now!
>> >>> http://p.sf.net/sfu/learndevnow-dev2
>> >>> _______________________________________________
>> >>> curl-loader-devel mailing list
>> >>> cur...@li...
>> >>> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel
>> >>>
>> >>
>> >>
>> >>
>> >> ------------------------------------------------------------------------------
>> >> Try before you buy = See our experts in action!
>> >> The most comprehensive online learning library for Microsoft developers
>> >> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3,
>> >> MVC3,
>> >> Metro Style Apps, more. Free future releases when you subscribe now!
>> >> http://p.sf.net/sfu/learndevnow-dev2
>> >> _______________________________________________
>> >> curl-loader-devel mailing list
>> >> cur...@li...
>> >> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel
>> >>
>> >
>> >
>> >
>> > --
>> > Regards,
>> > Robert Iakobashvili, Ph.D.
>> >
>> > Home: http://www.ghotit.com
>> > Blog: http://www.ghotit.com/category/ghotit-blogs/
>> > Twitter: http://twitter.com/ghotit
>> > Facebook: http://facebook.com/ghotit
>> > ......................................................................
>> > Ghotit Dyslexia
>> > Assistive technology that understands you
>> > ......................................................................
>>
>>
>>
>> --
>> Regards,
>> Robert Iakobashvili, Ph.D.
>>
>> Home: http://www.ghotit.com
>> Blog: http://www.ghotit.com/category/ghotit-blogs/
>> Twitter: http://twitter.com/ghotit
>> Facebook: http://facebook.com/ghotit
>> ......................................................................
>> Ghotit Dyslexia
>> Assistive technology that understands you
>> ......................................................................
>>
>>
>> ------------------------------------------------------------------------------
>> Try before you buy = See our experts in action!
>> The most comprehensive online learning library for Microsoft developers
>> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3,
>> Metro Style Apps, more. Free future releases when you subscribe now!
>> http://p.sf.net/sfu/learndevnow-dev2
>> _______________________________________________
>> curl-loader-devel mailing list
>> cur...@li...
>> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel
>
>
>
> ------------------------------------------------------------------------------
> Keep Your Developer Skills Current with LearnDevNow!
> The most comprehensive online learning library for Microsoft developers
> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3,
> Metro Style Apps, more. Free future releases when you subscribe now!
> http://p.sf.net/sfu/learndevnow-d2d
> _______________________________________________
> curl-loader-devel mailing list
> cur...@li...
> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel
>
--
Regards,
Robert Iakobashvili, Ph.D.
Home: http://www.ghotit.com
Blog: http://www.ghotit.com/category/ghotit-blogs/
Twitter: http://twitter.com/ghotit
Facebook: http://facebook.com/ghotit
......................................................................
Ghotit Dyslexia
Assistive technology that understands you
......................................................................
|
|
From: Fred H. <di...@gm...> - 2012-03-06 01:52:45
|
TIMEOUT -1 -> 60 works a little better but no performance improvement on
CAPS or throughput.
*test 5:*
enable c-ares;
curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, 60);
dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve *.com to
one IP address
number of domain names in URLs: 780,000
number of client: 1000
CPU usage: 74%
cpu% irq% sirq% sys% iowt% mem_used buf&cached
74.0 0.0 0.7 4.2 0.0 3530.6Mb 250.8Mb
SSL TPS: 360
SSL throughput: 220Mbps
# gprof /usr/bin/curl-loader gmon.out -p | head -50
Flat profile:
Each sample counts as 0.01 seconds.
% cumulative self self total
time seconds seconds calls s/call s/call name
26.63 23.13 23.13 119226 0.00 0.00
Curl_hash_clean_with_criterium
20.34 40.80 17.67 317792146 0.00 0.00
hostcache_timestamp_remove
11.27 50.59 9.79 2579903 0.00 0.00 Curl_hash_pick
7.84 57.40 6.81 120606 0.00 0.00 create_conn
5.65 62.31 4.91 119117401 0.00 0.00 Curl_str_key_compare
4.98 66.64 4.33 329829 0.00 0.00 Curl_hash_add
2.80 69.08 2.44 4476606 0.00 0.00 dprintf_formatf
2.57 71.31 2.23 124019 0.00 0.00 ConnectionStore
1.50 72.61 1.30 740997609 0.00 0.00 ares__is_list_empty
1.16 73.61 1.01 locking_function
0.76 74.27 0.66 238744 0.00 0.00 ares_cancel
0.70 74.88 0.61 457564 0.00 0.00
curl_multi_socket_action
0.67 75.46 0.58 118616 0.00 0.00
curl_multi_remove_handle
0.58 75.97 0.51 8211502 0.00 0.00
client_tracing_function
0.53 76.43 0.46 733561 0.00 0.00 Curl_readwrite
0.51 76.87 0.44 2183179 0.00 0.00 multi_runsingle
0.51 77.31 0.44 166585567 0.00 0.00 addbyter
0.44 77.69 0.38 2175805 0.00 0.00 Curl_raw_equal
0.43 78.06 0.38 id_function
0.40 78.41 0.35 29789346 0.00 0.00 Curl_socket_check
0.40 78.76 0.35 479671 0.00 0.00 ossl_connect_common
0.40 79.11 0.35 490639 0.00 0.00 ares__init_list_node
0.33 79.40 0.29 94197670 0.00 0.00 Curl_raw_toupper
0.33 79.69 0.29 3268031 0.00 0.00 Curl_pgrsUpdate
0.28 79.93 0.24 8797 0.00 0.00 curl_multi_perform
0.27 80.17 0.24 2233641 0.00 0.00 Curl_infof
0.26 80.40 0.23 1839837 0.00 0.00 Curl_splay
0.24 80.61 0.21 121713028 0.00 0.00 curl_strequal
0.21 80.79 0.18 243333 0.00 0.00 Curl_ssl_getsessionid
0.18 80.95 0.16 122180 0.00 0.00 Curl_num_addresses
0.17 81.09 0.15 8271666 0.00 0.00 Curl_debug
0.16 81.23 0.14 18902452 0.00 0.00 alloc_addbyter
0.16 81.37 0.14 8851 0.00 0.00 curl_multi_socket_all
0.14 81.50 0.13 2236769 0.00 0.00 Curl_read
0.14 81.62 0.12 2271700 0.00 0.00 fd_key_compare
0.14 81.74 0.12 323167 0.00 0.00 event_del
0.14 81.86 0.12 205436 0.00 0.00 read_udp_packets
0.14 81.98 0.12 200979 0.00 0.00 processfds
0.13 82.09 0.11 123886 0.00 0.00
Curl_if_is_interface_name
0.13 82.20 0.11 120350 0.00 0.00 Curl_http
0.13 82.31 0.11 ssl_tls_trace
0.12 82.41 0.10 119229 0.00 0.00 Curl_splayremovebyaddr
0.12 82.51 0.10 7798717 0.00 0.00 Curl_raw_nequal
0.12 82.61 0.10 784485 0.00 0.00 event_add
0.10 82.70 0.09 643482 0.00 0.00 Curl_expire
*test 6:*
enable c-ares;
curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, 60);
CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.23.0
dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve *.com to
one IP address
number of domain names in URLs: 780,000
number of client: 1000
CPU usage: 86%
cpu% irq% sirq% sys% iowt% mem_used buf&cached
85.9 0.0 0.2 2.1 0.0 3537.2Mb 261.4Mb
SSL TPS: 360
SSL throughput: 200Mbps
# gprof /usr/bin/curl-loader gmon.out -p | head -20
Flat profile:
Each sample counts as 0.01 seconds.
% cumulative self self total
time seconds seconds calls s/call s/call name
27.19 19.96 19.96 103472 0.00 0.00
Curl_hash_clean_with_criterium
21.73 35.91 15.95 274168981 0.00 0.00
hostcache_timestamp_remove
10.33 43.49 7.58 2256441 0.00 0.00 Curl_hash_pick
7.21 48.78 5.29 104540 0.00 0.00 create_conn
6.20 53.33 4.55 103413405 0.00 0.00 Curl_str_key_compare
4.88 56.91 3.58 287109 0.00 0.00 Curl_hash_add
3.02 59.13 2.22 3797855 0.00 0.00 dprintf_formatf
2.10 60.67 1.54 107711 0.00 0.00 ConnectionStore
1.65 61.88 1.21 642725327 0.00 0.00 ares__is_list_empty
1.19 62.76 0.88 locking_function
0.87 63.40 0.64 207817 0.00 0.00 ares_cancel
0.56 63.81 0.41 404108 0.00 0.00
curl_multi_socket_action
0.54 64.21 0.40 425603 0.00 0.00 ares__init_list_node
0.53 64.60 0.39 7068112 0.00 0.00
client_tracing_function
0.49 64.96 0.36 1908587 0.00 0.00 multi_runsingle
*test 7:*
enable c-ares;
curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, -1);
CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.23.0
dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve *.com to
one IP address
number of domain names in URLs: 780,000
number of client: 1000
CPU usage: 86%
cpu% irq% sirq% sys% iowt% mem_used buf&cached
86.6 0.0 0.6 3.3 0.0 3624.8Mb 263.0Mb
SSL TPS: 300
SSL throughput: 200Mbps
# gprof /usr/bin/curl-loader gmon.out -p | head -20
Flat profile:
Each sample counts as 0.01 seconds.
% cumulative self self total
time seconds seconds calls s/call s/call name
39.83 369.71 369.71 207173 0.00 0.00
Curl_hash_clean_with_criterium
24.27 594.98 225.27 3198337733 0.00 0.00
hostcache_timestamp_remove
13.75 722.59 127.61 4312864 0.00 0.00 Curl_hash_pick
9.19 807.87 85.28 546507 0.00 0.00 Curl_hash_add
6.14 864.88 57.01 1022419684 0.00 0.00 Curl_str_key_compare
2.06 884.00 19.12 203749 0.00 0.00 create_conn
0.54 888.99 4.99 213306 0.00 0.00 ConnectionStore
0.49 893.50 4.51 7331251 0.00 0.00 dprintf_formatf
0.29 896.16 2.66 1287182329 0.00 0.00 ares__is_list_empty
0.23 898.28 2.12 locking_function
0.20 900.10 1.82 212210 0.00 0.00 Curl_num_addresses
0.17 901.66 1.56 198385 0.00 0.00
curl_multi_remove_handle
0.12 902.81 1.15 411114 0.00 0.00 ares_cancel
0.12 903.95 1.14 3683448 0.00 0.00 multi_runsingle
0.11 904.97 1.03 13664120 0.00 0.00
client_tracing_function
*test 8:*
enable c-ares;
curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, 60);
CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.24.0
dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve *.com to
one IP address
number of domain names in URLs: 780,000
number of client: 1000
CPU usage: 75%
cpu% irq% sirq% sys% iowt% mem_used buf&cached
75.2 0.0 0.2 2.1 0.0 3537.2Mb 261.4Mb
SSL TPS: 380
SSL throughput: 220Mbps
# gprof /usr/bin/curl-loader gmon.out -p | head -20
Flat profile:
Each sample counts as 0.01 seconds.
% cumulative self self total
time seconds seconds calls s/call s/call name
27.58 12.31 12.31 71140 0.00 0.00
Curl_hash_clean_with_criterium
19.09 20.83 8.52 176729692 0.00 0.00
hostcache_timestamp_remove
10.44 25.49 4.66 1589309 0.00 0.00 Curl_hash_pick
6.85 28.55 3.06 73204 0.00 0.00 create_conn
5.56 31.03 2.48 67620530 0.00 0.00 Curl_str_key_compare
4.41 33.00 1.97 200684 0.00 0.00 Curl_hash_add
3.56 34.59 1.59 2717956 0.00 0.00 dprintf_formatf
1.72 35.36 0.77 75247 0.00 0.00 ConnectionStore
1.70 36.12 0.76 444342256 0.00 0.00 ares__is_list_empty
1.23 36.67 0.55 locking_function
1.16 37.19 0.52 143342 0.00 0.00 ares_cancel
0.99 37.63 0.44 72211 0.00 0.00
curl_multi_remove_handle
0.74 37.96 0.33 1333371 0.00 0.00 multi_runsingle
0.68 38.27 0.31 101147798 0.00 0.00 addbyter
0.58 38.53 0.26 288737 0.00 0.00
curl_multi_socket_action
*test 9:*
enable c-ares;
curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, -1);
CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.24.0
dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve *.com to
one IP address
number of domain names in URLs: 780,000
number of client: 1000
CPU usage: 75%
cpu% irq% sirq% sys% iowt% mem_used buf&cached
75.1 0.0 0.6 4.1 0.0 3545.2Mb 260.1Mb
SSL TPS: 380
SSL throughput: 200Mbps
# gprof /usr/bin/curl-loader gmon.out -p | head -20
Flat profile:
Each sample counts as 0.01 seconds.
% cumulative self self total
time seconds seconds calls s/call s/call name
34.00 31.13 31.13 72495 0.00 0.00
Curl_hash_clean_with_criterium
23.64 52.78 21.65 386070412 0.00 0.00
hostcache_timestamp_remove
13.28 64.94 12.16 1619440 0.00 0.00 Curl_hash_pick
5.92 70.36 5.42 137653129 0.00 0.00 Curl_str_key_compare
5.50 75.40 5.04 203018 0.00 0.00 Curl_hash_add
4.55 79.57 4.17 74618 0.00 0.00 create_conn
1.70 81.13 1.56 2757148 0.00 0.00 dprintf_formatf
0.98 82.03 0.90 76945 0.00 0.00 ConnectionStore
0.95 82.90 0.87 455775100 0.00 0.00 ares__is_list_empty
0.57 83.42 0.53 locking_function
0.39 83.78 0.36 146742 0.00 0.00 ares_cancel
0.35 84.10 0.32 102968806 0.00 0.00 addbyter
0.35 84.42 0.32 456587 0.00 0.00 Curl_readwrite
0.35 84.74 0.32 73133 0.00 0.00
curl_multi_remove_handle
0.34 85.05 0.31 1360545 0.00 0.00 multi_runsingle
2012/3/6 Robert Iakobashvili <cor...@gm...>
> If it will not be helpful this means that recent
> changes in libcurl library have broken DNS interface.
>
> You can try a previos version and the latest libcurl, like:
>
> http://curl.haxx.se/download/curl-7.23.0.tar.gz
> and
> http://curl.haxx.se/downloadcurl-7.24.0.tar.gz
>
> by:
>
> 1. downloading a libcurl version and placing it to
> curl-loader-0.56/packages ;
> 2. correcting the Makefile in curl-loader-0.56
> CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.24 or CURL_VER:=7.23
> 3. make cleanall; make
>
> Please, send us your observations.
> Thanks!
> Robert
>
> On Mon, Mar 5, 2012 at 6:57 PM, Robert Iakobashvili <cor...@gm...>
> wrote:
> > If so, please try to change
> >
> > loader.c, line 578 which looks like:
> > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, -1);
> >
> > 1. instead of -1 try to set some value like 60
> > 2. make clean; make
> >
> > Thanks,
> > Robert
> >
> > On Mon, Mar 5, 2012 at 6:19 PM, Fred Huang <di...@gm...> wrote:
> >>
> >> Looks like the bottleneck comes from libcurl other than c-ares...
> >> test 3 (no ares patch applied):
> >> dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve
> *.com to one IP address
> >> number of domain names in URLs: 780,000
> >> number of client: 1000
> >> CPU usage: 85%
> >> cpu% irq% sirq% sys% iowt% mem_used buf&cached
> >> 85.6 0.0 0.8 5.9 0.0 3334.3Mb 112.0Mb
> >> SSL TPS: 320
> >> SSL throughput: 200Mbps
> >>
> >> # gprof /usr/bin/curl-loader gmon.out -p | head -50
> >> Flat profile:
> >> Each sample counts as 0.01 seconds.
> >> % cumulative self self total
> >> time seconds seconds calls s/call s/call name
> >> 40.80 120.16 120.16 116079 0.00 0.00
> Curl_hash_clean_with_criterium
> >> 24.46 192.18 72.02 983166323 0.00 0.00
> hostcache_timestamp_remove
> >> 13.30 231.35 39.17 2059341 0.00 0.00 Curl_hash_pick
> >> 6.45 250.35 19.00 240087 0.00 0.00 Curl_hash_add
> >> 4.81 264.51 14.16 334257102 0.00 0.00
> Curl_str_key_compare
> >> 3.40 274.52 10.01 119514 0.00 0.00 create_conn
> >> 0.93 277.25 2.73 4478910 0.00 0.00 dprintf_formatf
> >> 0.90 279.91 2.66 122209 0.00 0.00 ConnectionStore
> >> 0.40 281.09 1.18 locking_function
> >> 0.27 281.88 0.79 114759 0.00 0.00
> curl_multi_remove_handle
> >> 0.21 282.51 0.63 7993664 0.00 0.00
> client_tracing_function
> >> 0.21 283.12 0.61 122685 0.00 0.00 Curl_num_addresses
> >> 0.19 283.68 0.56 397028 0.00 0.00
> curl_multi_socket_action
> >> 0.19 284.23 0.55 2036116 0.00 0.00 multi_runsingle
> >> 0.17 284.72 0.49 703687 0.00 0.00 Curl_readwrite
> >> 0.16 285.19 0.47 id_function
> >> 0.15 285.62 0.43 29692029 0.00 0.00 Curl_socket_check
> >> 0.14 286.02 0.40 91973856 0.00 0.00 Curl_raw_toupper
> >> 0.13 286.40 0.38 164664144 0.00 0.00 addbyter
> >> 0.13 286.77 0.37 2180797 0.00 0.00 Curl_infof
> >> 0.11 287.10 0.33 122675 0.00 0.00
> Curl_if_is_interface_name
> >> 0.10 287.39 0.29 471318 0.00 0.00
> ossl_connect_common
> >> 0.10 287.67 0.28 2113528 0.00 0.00 Curl_raw_equal
> >> 0.09 287.93 0.26 8676 0.00 0.02
> curl_multi_socket_all
> >> 0.08 288.17 0.24 1663262 0.00 0.00 Curl_splay
> >> 0.08 288.41 0.24 8505 0.00 0.01 curl_multi_perform
> >> 0.07 288.63 0.22 3114576 0.00 0.00 Curl_pgrsUpdate
> >> 0.06 288.80 0.17 119637 0.00 0.00 singleipconnect
> >> 0.06 288.97 0.17 7994818 0.00 0.00 Curl_debug
> >> 0.05 289.13 0.16 235820 0.00 0.00 event_del
> >> 0.05 289.29 0.16 119393071 0.00 0.00 curl_strequal
> >> 0.05 289.45 0.16 18535473 0.00 0.00 alloc_addbyter
> >> 0.05 289.61 0.16 356074 0.00 0.00 Curl_hash_str
> >> 0.05 289.76 0.15 1797233 0.00 0.00 fd_key_compare
> >> 0.05 289.91 0.15 234966 0.00 0.00
> Curl_ssl_getsessionid
> >> 0.04 290.04 0.13 585853 0.00 0.00 Curl_llist_destroy
> >> 0.04 290.16 0.12 240826 0.00 0.00 Curl_resolv
> >> 0.03 290.26 0.10 7514328 0.00 0.00 Curl_raw_nequal
> >> 0.03 290.36 0.10 2214436 0.00 0.00 Curl_read
> >> 0.03 290.46 0.10 116872 0.00 0.00
> Curl_http_readwrite_headers
> >> 0.03 290.56 0.10 ssl_tls_trace
> >> 0.03 290.66 0.10 8013097 0.00 0.00 scan_response
> >> 0.03 290.75 0.09 29488555 0.00 0.00 curlx_tvdiff
> >> 0.03 290.84 0.09 2142468 0.00 0.00 Curl_setopt
> >> 0.03 290.93 0.09 476299 0.00 0.00 Curl_expire
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >> 2012/3/5 Robert Iakobashvili <cor...@gm...>
> >>>
> >>> Dear Fred Huang,
> >>>
> >>> Thanks for your posting.
> >>>
> >>> Could you please check if this is due to some badness at cares?
> >>> Thanks.
> >>>
> >>> Let's exclude usage of cares based DNS resolving by applying the patch
> attached
> >>> and use instead global DNS cache.
> >>>
> >>> cd ./curl-loader-0.56
> >>> patch -p1 < path-to/disable-cares-use-global-dns.patch
> >>> make cleanall; make
> >>>
> >>> See if it helps.
> >>>
> >>> Global cache, unfortunately, is not thread safe.
> >>>
> >>> If it helps, we'll try to restore the previous version of cares.
> >>>
> >>> Thanks,
> >>> Robert
> >>>
> >>> On Mon, Mar 5, 2012 at 4:28 PM, Fred Huang <di...@gm...> wrote:
> >>>>
> >>>> test 1:
> >>>>
> >>>> dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve
> *.com to one IP address
> >>>> number of domain names in all https URLs: 780,000
> >>>> number of client: 1000
> >>>> CPU usage: 78%
> >>>> cpu% irq% sirq% sys% iowt% mem_used buf&cached
> >>>> 78.8 0.0 0.6 3.8 0.0 3640.9Mb 282.1Mb
> >>>> SSL TPS: 300
> >>>> SSL throughput: 200Mbps
> >>>>
> >>>> # gprof /usr/bin/curl-loader gmon.out -p | head -50
> >>>> Flat profile:
> >>>> Each sample counts as 0.01 seconds.
> >>>> % cumulative self self total
> >>>> time seconds seconds calls s/call s/call name
> >>>> 38.55 127.10 127.10 130975 0.00 0.00
> Curl_hash_clean_with_criterium
> >>>> 24.63 208.32 81.22 1247528407 0.00 0.00
> hostcache_timestamp_remove
> >>>> 13.08 251.44 43.12 2791988 0.00 0.00 Curl_hash_pick
> >>>> 7.30 275.51 24.07 352490 0.00 0.00 Curl_hash_add
> >>>> 6.01 295.31 19.80 410760770 0.00 0.00
> Curl_str_key_compare
> >>>> 2.95 305.04 9.73 131015 0.00 0.00 create_conn
> >>>> 0.88 307.94 2.90 4841390 0.00 0.00 dprintf_formatf
> >>>> 0.74 310.38 2.44 136036 0.00 0.00 ConnectionStore
> >>>> 0.48 311.96 1.58 814604911 0.00 0.00
> ares__is_list_empty
> >>>> 0.36 313.16 1.20 locking_function
> >>>> 0.27 314.06 0.90 261608 0.00 0.00 ares_cancel
> >>>> 0.22 314.79 0.73 127417 0.00 0.00
> curl_multi_remove_handle
> >>>> 0.19 315.40 0.62 8906301 0.00 0.00
> client_tracing_function
> >>>> 0.17 315.97 0.57 135305 0.00 0.00
> Curl_num_addresses
> >>>> 0.17 316.54 0.57 181937907 0.00 0.00 addbyter
> >>>> 0.17 317.09 0.55 2374363 0.00 0.00 multi_runsingle
> >>>> 0.16 317.62 0.53 541954 0.00 0.00
> ares__init_list_node
> >>>> 0.16 318.14 0.52 494412 0.00 0.00
> curl_multi_socket_action
> >>>> 0.14 318.60 0.46 33578897 0.00 0.00
> Curl_socket_check
> >>>> 0.13 319.04 0.44 102951052 0.00 0.00
> Curl_raw_toupper
> >>>> 0.13 319.48 0.44 787558 0.00 0.00 Curl_readwrite
> >>>> 0.13 319.91 0.44 id_function
> >>>> 0.12 320.29 0.38 391390 0.00 0.00 Curl_hash_str
> >>>> 0.09 320.58 0.29 9216 0.00 0.00
> curl_multi_perform
> >>>> 0.08 320.86 0.28 2350908 0.00 0.00 Curl_raw_equal
> >>>> 0.08 321.14 0.28 2019415 0.00 0.00 Curl_splay
> >>>> 0.08 321.40 0.26 520625 0.00 0.00
> ossl_connect_common
> >>>> 0.08 321.65 0.25 3532992 0.00 0.00 Curl_pgrsUpdate
> >>>> 0.07 321.87 0.22 131778537 0.00 0.00 curl_strequal
> >>>> 0.05 322.05 0.18 8954118 0.00 0.00 scan_response
> >>>> 0.05 322.23 0.18 533644 0.00 0.00 ares_expand_name
> >>>> 0.05 322.40 0.17 2477345 0.00 0.00 fd_key_compare
> >>>> 0.05 322.57 0.17 2429135 0.00 0.00 Curl_infof
> >>>> 0.05 322.74 0.17 130966 0.00 0.00 singleipconnect
> >>>> 0.05 322.90 0.16 9292 0.00 0.00
> curl_multi_socket_all
> >>>> 0.05 323.06 0.16 270137 0.00 0.00
> ares__get_hostent
> >>>> 0.05 323.21 0.15 8943657 0.00 0.00 Curl_debug
> >>>> 0.04 323.35 0.14 262493 0.00 0.00
> Curl_ssl_getsessionid
> >>>> 0.04 323.48 0.13 566006 0.00 0.00 socket_callback
> >>>> 0.04 323.61 0.13 33983864 0.00 0.00 curlx_tvdiff
> >>>> 0.04 323.74 0.13 132853 0.00 0.00
> Curl_http_readwrite_headers
> >>>> 0.04 323.86 0.12 344032 0.00 0.00 epoll_del
> >>>> 0.04 323.98 0.12 223477 0.00 0.00 Curl_poll
> >>>> 0.04 324.10 0.12 8526280 0.00 0.00 Curl_raw_nequal
> >>>> 0.03 324.21 0.11 348839 0.00 0.00 event_del
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> test 2:
> >>>>
> >>>> dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve
> *.com to one IP address
> >>>> number of domain names in all https URLs: 1
> >>>> number of client: 1000
> >>>> CPU usage: 75%
> >>>> SSL TPS: 1300
> >>>> SSL throughput: 700Mbps
> >>>>
> >>>> # gprof /usr/bin/curl-loader gmon.out -p | head -50
> >>>> Flat profile:
> >>>> Each sample counts as 0.01 seconds.
> >>>> % cumulative self self total
> >>>> time seconds seconds calls s/call s/call name
> >>>> 8.98 2.45 2.45 5127312 0.00 0.00 dprintf_formatf
> >>>> 8.90 4.88 2.43 1292796503 0.00 0.00
> ares__is_list_empty
> >>>> 6.12 6.55 1.67 322461 0.00 0.00 create_conn
> >>>> 4.58 7.80 1.25 419872 0.00 0.00 ares_cancel
> >>>> 4.10 8.92 1.12 2100162 0.00 0.00 Curl_readwrite
> >>>> 3.92 9.99 1.07 4527388 0.00 0.00 Curl_hash_pick
> >>>> 3.43 10.93 0.94 locking_function
> >>>> 3.04 11.76 0.83 909759 0.00 0.00
> curl_multi_socket_action
> >>>> 3.00 12.58 0.82 97252 0.00 0.00
> Curl_hash_clean_with_criterium
> >>>> 2.71 13.32 0.74 179704016 0.00 0.00
> Curl_raw_toupper
> >>>> 2.68 14.05 0.73 4646524 0.00 0.00 multi_runsingle
> >>>> 2.44 14.71 0.67 13862992 0.00 0.00
> client_tracing_function
> >>>> 2.42 15.37 0.66 319735 0.00 0.00
> curl_multi_remove_handle
> >>>> 2.42 16.03 0.66 22 0.03 0.03
> ares__init_list_node
> >>>> 1.94 16.56 0.53 9930244 0.00 0.00
> hostcache_timestamp_remove
> >>>> 1.94 17.09 0.53 169320088 0.00 0.00 addbyter
> >>>> 1.80 17.58 0.49 id_function
> >>>> 1.36 17.95 0.37 7057146 0.00 0.00 Curl_pgrsUpdate
> >>>> 1.36 18.32 0.37 16835 0.00 0.00
> curl_multi_socket_all
> >>>> 1.25 18.66 0.34 16719 0.00 0.00
> curl_multi_perform
> >>>> 1.21 18.99 0.33 329403 0.00 0.00
> Curl_http_readwrite_headers
> >>>> 1.17 19.31 0.32 4036624 0.00 0.00 Curl_splay
> >>>> 1.14 19.62 0.31 20838902 0.00 0.00 Curl_raw_nequal
> >>>> 0.82 19.85 0.23 2764123 0.00 0.00 Curl_infof
> >>>> 0.81 20.07 0.22 5933857 0.00 0.00 ossl_recv
> >>>> 0.77 20.28 0.21 323112 0.00 0.00
> Curl_splayremovebyaddr
> >>>> 0.75 20.48 0.21 5943991 0.00 0.00 Curl_read
> >>>> 0.70 20.67 0.19 425174 0.00 0.00 event_del
> >>>> 0.70 20.86 0.19 98652 0.00 0.00
> Curl_if_is_interface_name
> >>>> 0.70 21.05 0.19 8677733 0.00 0.00
> Curl_socket_check
> >>>> 0.59 21.21 0.16 4129554 0.00 0.00 fd_key_compare
> >>>> 0.57 21.37 0.16 14017019 0.00 0.00 Curl_debug
> >>>> 0.55 21.52 0.15 9376861 0.00 0.00 stat_data_in_add
> >>>> 0.55 21.67 0.15 6840468 0.00 0.00 Curl_timeleft
> >>>> 0.55 21.82 0.15 2054602 0.00 0.00 Curl_raw_equal
> >>>> 0.55 21.97 0.15 1291611 0.00 0.00 Curl_expire
> >>>> 0.51 22.11 0.14 16570063 0.00 0.00 curlx_tvnow
> >>>> 0.51 22.25 0.14 13863284 0.00 0.00 scan_response
> >>>> 0.51 22.39 0.14 5498864 0.00 0.00 Curl_setopt
> >>>> 0.48 22.52 0.13 97490291 0.00 0.00 curl_strequal
> >>>> 0.44 22.64 0.12 324253 0.00 0.00 Curl_http
> >>>> 0.44 22.76 0.12 394386 0.00 0.00
> ossl_connect_common
> >>>> 0.40 22.87 0.11 3581335 0.00 0.00 Curl_getinfo
> >>>> 0.38 22.97 0.11 26983092 0.00 0.00 alloc_addbyter
> >>>> 0.37 23.07 0.10 8729044 0.00 0.00
> Curl_client_write
> >>>>
> >>>>
> >>>>
> ------------------------------------------------------------------------------
> >>>> Try before you buy = See our experts in action!
> >>>> The most comprehensive online learning library for Microsoft
> developers
> >>>> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3,
> MVC3,
> >>>> Metro Style Apps, more. Free future releases when you subscribe now!
> >>>> http://p.sf.net/sfu/learndevnow-dev2
> >>>> _______________________________________________
> >>>> curl-loader-devel mailing list
> >>>> cur...@li...
> >>>> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> Regards,
> >>> Robert Iakobashvili, Ph.D.
> >>>
> >>> Home: http://www.ghotit.com
> >>> Blog: http://www.ghotit.com/category/ghotit-blogs/
> >>> Twitter: http://twitter.com/ghotit
> >>> Facebook: http://facebook.com/ghotit
> >>> ......................................................................
> >>> Ghotit Dyslexia
> >>> Assistive technology that understands you
> >>> ......................................................................
> >>>
> >>>
> >>>
> ------------------------------------------------------------------------------
> >>> Try before you buy = See our experts in action!
> >>> The most comprehensive online learning library for Microsoft developers
> >>> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3,
> MVC3,
> >>> Metro Style Apps, more. Free future releases when you subscribe now!
> >>> http://p.sf.net/sfu/learndevnow-dev2
> >>> _______________________________________________
> >>> curl-loader-devel mailing list
> >>> cur...@li...
> >>> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel
> >>>
> >>
> >>
> >>
> ------------------------------------------------------------------------------
> >> Try before you buy = See our experts in action!
> >> The most comprehensive online learning library for Microsoft developers
> >> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3,
> >> Metro Style Apps, more. Free future releases when you subscribe now!
> >> http://p.sf.net/sfu/learndevnow-dev2
> >> _______________________________________________
> >> curl-loader-devel mailing list
> >> cur...@li...
> >> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel
> >>
> >
> >
> >
> > --
> > Regards,
> > Robert Iakobashvili, Ph.D.
> >
> > Home: http://www.ghotit.com
> > Blog: http://www.ghotit.com/category/ghotit-blogs/
> > Twitter: http://twitter.com/ghotit
> > Facebook: http://facebook.com/ghotit
> > ......................................................................
> > Ghotit Dyslexia
> > Assistive technology that understands you
> > ......................................................................
>
>
>
> --
> Regards,
> Robert Iakobashvili, Ph.D.
>
> Home: http://www.ghotit.com
> Blog: http://www.ghotit.com/category/ghotit-blogs/
> Twitter: http://twitter.com/ghotit
> Facebook: http://facebook.com/ghotit
> ......................................................................
> Ghotit Dyslexia
> Assistive technology that understands you
> ......................................................................
>
>
> ------------------------------------------------------------------------------
> Try before you buy = See our experts in action!
> The most comprehensive online learning library for Microsoft developers
> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3,
> Metro Style Apps, more. Free future releases when you subscribe now!
> http://p.sf.net/sfu/learndevnow-dev2
> _______________________________________________
> curl-loader-devel mailing list
> cur...@li...
> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel
>
|
|
From: Robert I. <cor...@gm...> - 2012-03-05 17:12:12
|
If it will not be helpful this means that recent changes in libcurl library have broken DNS interface. You can try a previos version and the latest libcurl, like: http://curl.haxx.se/download/curl-7.23.0.tar.gz and http://curl.haxx.se/downloadcurl-7.24.0.tar.gz by: 1. downloading a libcurl version and placing it to curl-loader-0.56/packages ; 2. correcting the Makefile in curl-loader-0.56 CURL_VER:=7.24.0-20120109 -> CURL_VER:=7.24 or CURL_VER:=7.23 3. make cleanall; make Please, send us your observations. Thanks! Robert On Mon, Mar 5, 2012 at 6:57 PM, Robert Iakobashvili <cor...@gm...> wrote: > If so, please try to change > > loader.c, line 578 which looks like: > curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, -1); > > 1. instead of -1 try to set some value like 60 > 2. make clean; make > > Thanks, > Robert > > On Mon, Mar 5, 2012 at 6:19 PM, Fred Huang <di...@gm...> wrote: >> >> Looks like the bottleneck comes from libcurl other than c-ares... >> test 3 (no ares patch applied): >> dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve *.com to one IP address >> number of domain names in URLs: 780,000 >> number of client: 1000 >> CPU usage: 85% >> cpu% irq% sirq% sys% iowt% mem_used buf&cached >> 85.6 0.0 0.8 5.9 0.0 3334.3Mb 112.0Mb >> SSL TPS: 320 >> SSL throughput: 200Mbps >> >> # gprof /usr/bin/curl-loader gmon.out -p | head -50 >> Flat profile: >> Each sample counts as 0.01 seconds. >> % cumulative self self total >> time seconds seconds calls s/call s/call name >> 40.80 120.16 120.16 116079 0.00 0.00 Curl_hash_clean_with_criterium >> 24.46 192.18 72.02 983166323 0.00 0.00 hostcache_timestamp_remove >> 13.30 231.35 39.17 2059341 0.00 0.00 Curl_hash_pick >> 6.45 250.35 19.00 240087 0.00 0.00 Curl_hash_add >> 4.81 264.51 14.16 334257102 0.00 0.00 Curl_str_key_compare >> 3.40 274.52 10.01 119514 0.00 0.00 create_conn >> 0.93 277.25 2.73 4478910 0.00 0.00 dprintf_formatf >> 0.90 279.91 2.66 122209 0.00 0.00 ConnectionStore >> 0.40 281.09 1.18 locking_function >> 0.27 281.88 0.79 114759 0.00 0.00 curl_multi_remove_handle >> 0.21 282.51 0.63 7993664 0.00 0.00 client_tracing_function >> 0.21 283.12 0.61 122685 0.00 0.00 Curl_num_addresses >> 0.19 283.68 0.56 397028 0.00 0.00 curl_multi_socket_action >> 0.19 284.23 0.55 2036116 0.00 0.00 multi_runsingle >> 0.17 284.72 0.49 703687 0.00 0.00 Curl_readwrite >> 0.16 285.19 0.47 id_function >> 0.15 285.62 0.43 29692029 0.00 0.00 Curl_socket_check >> 0.14 286.02 0.40 91973856 0.00 0.00 Curl_raw_toupper >> 0.13 286.40 0.38 164664144 0.00 0.00 addbyter >> 0.13 286.77 0.37 2180797 0.00 0.00 Curl_infof >> 0.11 287.10 0.33 122675 0.00 0.00 Curl_if_is_interface_name >> 0.10 287.39 0.29 471318 0.00 0.00 ossl_connect_common >> 0.10 287.67 0.28 2113528 0.00 0.00 Curl_raw_equal >> 0.09 287.93 0.26 8676 0.00 0.02 curl_multi_socket_all >> 0.08 288.17 0.24 1663262 0.00 0.00 Curl_splay >> 0.08 288.41 0.24 8505 0.00 0.01 curl_multi_perform >> 0.07 288.63 0.22 3114576 0.00 0.00 Curl_pgrsUpdate >> 0.06 288.80 0.17 119637 0.00 0.00 singleipconnect >> 0.06 288.97 0.17 7994818 0.00 0.00 Curl_debug >> 0.05 289.13 0.16 235820 0.00 0.00 event_del >> 0.05 289.29 0.16 119393071 0.00 0.00 curl_strequal >> 0.05 289.45 0.16 18535473 0.00 0.00 alloc_addbyter >> 0.05 289.61 0.16 356074 0.00 0.00 Curl_hash_str >> 0.05 289.76 0.15 1797233 0.00 0.00 fd_key_compare >> 0.05 289.91 0.15 234966 0.00 0.00 Curl_ssl_getsessionid >> 0.04 290.04 0.13 585853 0.00 0.00 Curl_llist_destroy >> 0.04 290.16 0.12 240826 0.00 0.00 Curl_resolv >> 0.03 290.26 0.10 7514328 0.00 0.00 Curl_raw_nequal >> 0.03 290.36 0.10 2214436 0.00 0.00 Curl_read >> 0.03 290.46 0.10 116872 0.00 0.00 Curl_http_readwrite_headers >> 0.03 290.56 0.10 ssl_tls_trace >> 0.03 290.66 0.10 8013097 0.00 0.00 scan_response >> 0.03 290.75 0.09 29488555 0.00 0.00 curlx_tvdiff >> 0.03 290.84 0.09 2142468 0.00 0.00 Curl_setopt >> 0.03 290.93 0.09 476299 0.00 0.00 Curl_expire >> >> >> >> >> >> >> >> >> 2012/3/5 Robert Iakobashvili <cor...@gm...> >>> >>> Dear Fred Huang, >>> >>> Thanks for your posting. >>> >>> Could you please check if this is due to some badness at cares? >>> Thanks. >>> >>> Let's exclude usage of cares based DNS resolving by applying the patch attached >>> and use instead global DNS cache. >>> >>> cd ./curl-loader-0.56 >>> patch -p1 < path-to/disable-cares-use-global-dns.patch >>> make cleanall; make >>> >>> See if it helps. >>> >>> Global cache, unfortunately, is not thread safe. >>> >>> If it helps, we'll try to restore the previous version of cares. >>> >>> Thanks, >>> Robert >>> >>> On Mon, Mar 5, 2012 at 4:28 PM, Fred Huang <di...@gm...> wrote: >>>> >>>> test 1: >>>> >>>> dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve *.com to one IP address >>>> number of domain names in all https URLs: 780,000 >>>> number of client: 1000 >>>> CPU usage: 78% >>>> cpu% irq% sirq% sys% iowt% mem_used buf&cached >>>> 78.8 0.0 0.6 3.8 0.0 3640.9Mb 282.1Mb >>>> SSL TPS: 300 >>>> SSL throughput: 200Mbps >>>> >>>> # gprof /usr/bin/curl-loader gmon.out -p | head -50 >>>> Flat profile: >>>> Each sample counts as 0.01 seconds. >>>> % cumulative self self total >>>> time seconds seconds calls s/call s/call name >>>> 38.55 127.10 127.10 130975 0.00 0.00 Curl_hash_clean_with_criterium >>>> 24.63 208.32 81.22 1247528407 0.00 0.00 hostcache_timestamp_remove >>>> 13.08 251.44 43.12 2791988 0.00 0.00 Curl_hash_pick >>>> 7.30 275.51 24.07 352490 0.00 0.00 Curl_hash_add >>>> 6.01 295.31 19.80 410760770 0.00 0.00 Curl_str_key_compare >>>> 2.95 305.04 9.73 131015 0.00 0.00 create_conn >>>> 0.88 307.94 2.90 4841390 0.00 0.00 dprintf_formatf >>>> 0.74 310.38 2.44 136036 0.00 0.00 ConnectionStore >>>> 0.48 311.96 1.58 814604911 0.00 0.00 ares__is_list_empty >>>> 0.36 313.16 1.20 locking_function >>>> 0.27 314.06 0.90 261608 0.00 0.00 ares_cancel >>>> 0.22 314.79 0.73 127417 0.00 0.00 curl_multi_remove_handle >>>> 0.19 315.40 0.62 8906301 0.00 0.00 client_tracing_function >>>> 0.17 315.97 0.57 135305 0.00 0.00 Curl_num_addresses >>>> 0.17 316.54 0.57 181937907 0.00 0.00 addbyter >>>> 0.17 317.09 0.55 2374363 0.00 0.00 multi_runsingle >>>> 0.16 317.62 0.53 541954 0.00 0.00 ares__init_list_node >>>> 0.16 318.14 0.52 494412 0.00 0.00 curl_multi_socket_action >>>> 0.14 318.60 0.46 33578897 0.00 0.00 Curl_socket_check >>>> 0.13 319.04 0.44 102951052 0.00 0.00 Curl_raw_toupper >>>> 0.13 319.48 0.44 787558 0.00 0.00 Curl_readwrite >>>> 0.13 319.91 0.44 id_function >>>> 0.12 320.29 0.38 391390 0.00 0.00 Curl_hash_str >>>> 0.09 320.58 0.29 9216 0.00 0.00 curl_multi_perform >>>> 0.08 320.86 0.28 2350908 0.00 0.00 Curl_raw_equal >>>> 0.08 321.14 0.28 2019415 0.00 0.00 Curl_splay >>>> 0.08 321.40 0.26 520625 0.00 0.00 ossl_connect_common >>>> 0.08 321.65 0.25 3532992 0.00 0.00 Curl_pgrsUpdate >>>> 0.07 321.87 0.22 131778537 0.00 0.00 curl_strequal >>>> 0.05 322.05 0.18 8954118 0.00 0.00 scan_response >>>> 0.05 322.23 0.18 533644 0.00 0.00 ares_expand_name >>>> 0.05 322.40 0.17 2477345 0.00 0.00 fd_key_compare >>>> 0.05 322.57 0.17 2429135 0.00 0.00 Curl_infof >>>> 0.05 322.74 0.17 130966 0.00 0.00 singleipconnect >>>> 0.05 322.90 0.16 9292 0.00 0.00 curl_multi_socket_all >>>> 0.05 323.06 0.16 270137 0.00 0.00 ares__get_hostent >>>> 0.05 323.21 0.15 8943657 0.00 0.00 Curl_debug >>>> 0.04 323.35 0.14 262493 0.00 0.00 Curl_ssl_getsessionid >>>> 0.04 323.48 0.13 566006 0.00 0.00 socket_callback >>>> 0.04 323.61 0.13 33983864 0.00 0.00 curlx_tvdiff >>>> 0.04 323.74 0.13 132853 0.00 0.00 Curl_http_readwrite_headers >>>> 0.04 323.86 0.12 344032 0.00 0.00 epoll_del >>>> 0.04 323.98 0.12 223477 0.00 0.00 Curl_poll >>>> 0.04 324.10 0.12 8526280 0.00 0.00 Curl_raw_nequal >>>> 0.03 324.21 0.11 348839 0.00 0.00 event_del >>>> >>>> >>>> >>>> >>>> test 2: >>>> >>>> dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve *.com to one IP address >>>> number of domain names in all https URLs: 1 >>>> number of client: 1000 >>>> CPU usage: 75% >>>> SSL TPS: 1300 >>>> SSL throughput: 700Mbps >>>> >>>> # gprof /usr/bin/curl-loader gmon.out -p | head -50 >>>> Flat profile: >>>> Each sample counts as 0.01 seconds. >>>> % cumulative self self total >>>> time seconds seconds calls s/call s/call name >>>> 8.98 2.45 2.45 5127312 0.00 0.00 dprintf_formatf >>>> 8.90 4.88 2.43 1292796503 0.00 0.00 ares__is_list_empty >>>> 6.12 6.55 1.67 322461 0.00 0.00 create_conn >>>> 4.58 7.80 1.25 419872 0.00 0.00 ares_cancel >>>> 4.10 8.92 1.12 2100162 0.00 0.00 Curl_readwrite >>>> 3.92 9.99 1.07 4527388 0.00 0.00 Curl_hash_pick >>>> 3.43 10.93 0.94 locking_function >>>> 3.04 11.76 0.83 909759 0.00 0.00 curl_multi_socket_action >>>> 3.00 12.58 0.82 97252 0.00 0.00 Curl_hash_clean_with_criterium >>>> 2.71 13.32 0.74 179704016 0.00 0.00 Curl_raw_toupper >>>> 2.68 14.05 0.73 4646524 0.00 0.00 multi_runsingle >>>> 2.44 14.71 0.67 13862992 0.00 0.00 client_tracing_function >>>> 2.42 15.37 0.66 319735 0.00 0.00 curl_multi_remove_handle >>>> 2.42 16.03 0.66 22 0.03 0.03 ares__init_list_node >>>> 1.94 16.56 0.53 9930244 0.00 0.00 hostcache_timestamp_remove >>>> 1.94 17.09 0.53 169320088 0.00 0.00 addbyter >>>> 1.80 17.58 0.49 id_function >>>> 1.36 17.95 0.37 7057146 0.00 0.00 Curl_pgrsUpdate >>>> 1.36 18.32 0.37 16835 0.00 0.00 curl_multi_socket_all >>>> 1.25 18.66 0.34 16719 0.00 0.00 curl_multi_perform >>>> 1.21 18.99 0.33 329403 0.00 0.00 Curl_http_readwrite_headers >>>> 1.17 19.31 0.32 4036624 0.00 0.00 Curl_splay >>>> 1.14 19.62 0.31 20838902 0.00 0.00 Curl_raw_nequal >>>> 0.82 19.85 0.23 2764123 0.00 0.00 Curl_infof >>>> 0.81 20.07 0.22 5933857 0.00 0.00 ossl_recv >>>> 0.77 20.28 0.21 323112 0.00 0.00 Curl_splayremovebyaddr >>>> 0.75 20.48 0.21 5943991 0.00 0.00 Curl_read >>>> 0.70 20.67 0.19 425174 0.00 0.00 event_del >>>> 0.70 20.86 0.19 98652 0.00 0.00 Curl_if_is_interface_name >>>> 0.70 21.05 0.19 8677733 0.00 0.00 Curl_socket_check >>>> 0.59 21.21 0.16 4129554 0.00 0.00 fd_key_compare >>>> 0.57 21.37 0.16 14017019 0.00 0.00 Curl_debug >>>> 0.55 21.52 0.15 9376861 0.00 0.00 stat_data_in_add >>>> 0.55 21.67 0.15 6840468 0.00 0.00 Curl_timeleft >>>> 0.55 21.82 0.15 2054602 0.00 0.00 Curl_raw_equal >>>> 0.55 21.97 0.15 1291611 0.00 0.00 Curl_expire >>>> 0.51 22.11 0.14 16570063 0.00 0.00 curlx_tvnow >>>> 0.51 22.25 0.14 13863284 0.00 0.00 scan_response >>>> 0.51 22.39 0.14 5498864 0.00 0.00 Curl_setopt >>>> 0.48 22.52 0.13 97490291 0.00 0.00 curl_strequal >>>> 0.44 22.64 0.12 324253 0.00 0.00 Curl_http >>>> 0.44 22.76 0.12 394386 0.00 0.00 ossl_connect_common >>>> 0.40 22.87 0.11 3581335 0.00 0.00 Curl_getinfo >>>> 0.38 22.97 0.11 26983092 0.00 0.00 alloc_addbyter >>>> 0.37 23.07 0.10 8729044 0.00 0.00 Curl_client_write >>>> >>>> >>>> ------------------------------------------------------------------------------ >>>> Try before you buy = See our experts in action! >>>> The most comprehensive online learning library for Microsoft developers >>>> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, >>>> Metro Style Apps, more. Free future releases when you subscribe now! >>>> http://p.sf.net/sfu/learndevnow-dev2 >>>> _______________________________________________ >>>> curl-loader-devel mailing list >>>> cur...@li... >>>> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel >>>> >>> >>> >>> >>> -- >>> Regards, >>> Robert Iakobashvili, Ph.D. >>> >>> Home: http://www.ghotit.com >>> Blog: http://www.ghotit.com/category/ghotit-blogs/ >>> Twitter: http://twitter.com/ghotit >>> Facebook: http://facebook.com/ghotit >>> ...................................................................... >>> Ghotit Dyslexia >>> Assistive technology that understands you >>> ...................................................................... >>> >>> >>> ------------------------------------------------------------------------------ >>> Try before you buy = See our experts in action! >>> The most comprehensive online learning library for Microsoft developers >>> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, >>> Metro Style Apps, more. Free future releases when you subscribe now! >>> http://p.sf.net/sfu/learndevnow-dev2 >>> _______________________________________________ >>> curl-loader-devel mailing list >>> cur...@li... >>> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel >>> >> >> >> ------------------------------------------------------------------------------ >> Try before you buy = See our experts in action! >> The most comprehensive online learning library for Microsoft developers >> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, >> Metro Style Apps, more. Free future releases when you subscribe now! >> http://p.sf.net/sfu/learndevnow-dev2 >> _______________________________________________ >> curl-loader-devel mailing list >> cur...@li... >> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel >> > > > > -- > Regards, > Robert Iakobashvili, Ph.D. > > Home: http://www.ghotit.com > Blog: http://www.ghotit.com/category/ghotit-blogs/ > Twitter: http://twitter.com/ghotit > Facebook: http://facebook.com/ghotit > ...................................................................... > Ghotit Dyslexia > Assistive technology that understands you > ...................................................................... -- Regards, Robert Iakobashvili, Ph.D. Home: http://www.ghotit.com Blog: http://www.ghotit.com/category/ghotit-blogs/ Twitter: http://twitter.com/ghotit Facebook: http://facebook.com/ghotit ...................................................................... Ghotit Dyslexia Assistive technology that understands you ...................................................................... |
|
From: Robert I. <cor...@gm...> - 2012-03-05 16:57:54
|
If so, please try to change loader.c, line 578 which looks like: curl_easy_setopt (handle, CURLOPT_DNS_CACHE_TIMEOUT, -1); 1. instead of -1 try to set some value like 60 2. make clean; make Thanks, Robert On Mon, Mar 5, 2012 at 6:19 PM, Fred Huang <di...@gm...> wrote: > > Looks like the bottleneck comes from libcurl other than c-ares... > test 3 (no ares patch applied): > dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve *.com to one IP address > number of domain names in URLs: 780,000 > number of client: 1000 > CPU usage: 85% > cpu% irq% sirq% sys% iowt% mem_used buf&cached > 85.6 0.0 0.8 5.9 0.0 3334.3Mb 112.0Mb > SSL TPS: 320 > SSL throughput: 200Mbps > > # gprof /usr/bin/curl-loader gmon.out -p | head -50 > Flat profile: > Each sample counts as 0.01 seconds. > % cumulative self self total > time seconds seconds calls s/call s/call name > 40.80 120.16 120.16 116079 0.00 0.00 Curl_hash_clean_with_criterium > 24.46 192.18 72.02 983166323 0.00 0.00 hostcache_timestamp_remove > 13.30 231.35 39.17 2059341 0.00 0.00 Curl_hash_pick > 6.45 250.35 19.00 240087 0.00 0.00 Curl_hash_add > 4.81 264.51 14.16 334257102 0.00 0.00 Curl_str_key_compare > 3.40 274.52 10.01 119514 0.00 0.00 create_conn > 0.93 277.25 2.73 4478910 0.00 0.00 dprintf_formatf > 0.90 279.91 2.66 122209 0.00 0.00 ConnectionStore > 0.40 281.09 1.18 locking_function > 0.27 281.88 0.79 114759 0.00 0.00 curl_multi_remove_handle > 0.21 282.51 0.63 7993664 0.00 0.00 client_tracing_function > 0.21 283.12 0.61 122685 0.00 0.00 Curl_num_addresses > 0.19 283.68 0.56 397028 0.00 0.00 curl_multi_socket_action > 0.19 284.23 0.55 2036116 0.00 0.00 multi_runsingle > 0.17 284.72 0.49 703687 0.00 0.00 Curl_readwrite > 0.16 285.19 0.47 id_function > 0.15 285.62 0.43 29692029 0.00 0.00 Curl_socket_check > 0.14 286.02 0.40 91973856 0.00 0.00 Curl_raw_toupper > 0.13 286.40 0.38 164664144 0.00 0.00 addbyter > 0.13 286.77 0.37 2180797 0.00 0.00 Curl_infof > 0.11 287.10 0.33 122675 0.00 0.00 Curl_if_is_interface_name > 0.10 287.39 0.29 471318 0.00 0.00 ossl_connect_common > 0.10 287.67 0.28 2113528 0.00 0.00 Curl_raw_equal > 0.09 287.93 0.26 8676 0.00 0.02 curl_multi_socket_all > 0.08 288.17 0.24 1663262 0.00 0.00 Curl_splay > 0.08 288.41 0.24 8505 0.00 0.01 curl_multi_perform > 0.07 288.63 0.22 3114576 0.00 0.00 Curl_pgrsUpdate > 0.06 288.80 0.17 119637 0.00 0.00 singleipconnect > 0.06 288.97 0.17 7994818 0.00 0.00 Curl_debug > 0.05 289.13 0.16 235820 0.00 0.00 event_del > 0.05 289.29 0.16 119393071 0.00 0.00 curl_strequal > 0.05 289.45 0.16 18535473 0.00 0.00 alloc_addbyter > 0.05 289.61 0.16 356074 0.00 0.00 Curl_hash_str > 0.05 289.76 0.15 1797233 0.00 0.00 fd_key_compare > 0.05 289.91 0.15 234966 0.00 0.00 Curl_ssl_getsessionid > 0.04 290.04 0.13 585853 0.00 0.00 Curl_llist_destroy > 0.04 290.16 0.12 240826 0.00 0.00 Curl_resolv > 0.03 290.26 0.10 7514328 0.00 0.00 Curl_raw_nequal > 0.03 290.36 0.10 2214436 0.00 0.00 Curl_read > 0.03 290.46 0.10 116872 0.00 0.00 Curl_http_readwrite_headers > 0.03 290.56 0.10 ssl_tls_trace > 0.03 290.66 0.10 8013097 0.00 0.00 scan_response > 0.03 290.75 0.09 29488555 0.00 0.00 curlx_tvdiff > 0.03 290.84 0.09 2142468 0.00 0.00 Curl_setopt > 0.03 290.93 0.09 476299 0.00 0.00 Curl_expire > > > > > > > > > 2012/3/5 Robert Iakobashvili <cor...@gm...> >> >> Dear Fred Huang, >> >> Thanks for your posting. >> >> Could you please check if this is due to some badness at cares? >> Thanks. >> >> Let's exclude usage of cares based DNS resolving by applying the patch attached >> and use instead global DNS cache. >> >> cd ./curl-loader-0.56 >> patch -p1 < path-to/disable-cares-use-global-dns.patch >> make cleanall; make >> >> See if it helps. >> >> Global cache, unfortunately, is not thread safe. >> >> If it helps, we'll try to restore the previous version of cares. >> >> Thanks, >> Robert >> >> On Mon, Mar 5, 2012 at 4:28 PM, Fred Huang <di...@gm...> wrote: >>> >>> test 1: >>> >>> dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve *.com to one IP address >>> number of domain names in all https URLs: 780,000 >>> number of client: 1000 >>> CPU usage: 78% >>> cpu% irq% sirq% sys% iowt% mem_used buf&cached >>> 78.8 0.0 0.6 3.8 0.0 3640.9Mb 282.1Mb >>> SSL TPS: 300 >>> SSL throughput: 200Mbps >>> >>> # gprof /usr/bin/curl-loader gmon.out -p | head -50 >>> Flat profile: >>> Each sample counts as 0.01 seconds. >>> % cumulative self self total >>> time seconds seconds calls s/call s/call name >>> 38.55 127.10 127.10 130975 0.00 0.00 Curl_hash_clean_with_criterium >>> 24.63 208.32 81.22 1247528407 0.00 0.00 hostcache_timestamp_remove >>> 13.08 251.44 43.12 2791988 0.00 0.00 Curl_hash_pick >>> 7.30 275.51 24.07 352490 0.00 0.00 Curl_hash_add >>> 6.01 295.31 19.80 410760770 0.00 0.00 Curl_str_key_compare >>> 2.95 305.04 9.73 131015 0.00 0.00 create_conn >>> 0.88 307.94 2.90 4841390 0.00 0.00 dprintf_formatf >>> 0.74 310.38 2.44 136036 0.00 0.00 ConnectionStore >>> 0.48 311.96 1.58 814604911 0.00 0.00 ares__is_list_empty >>> 0.36 313.16 1.20 locking_function >>> 0.27 314.06 0.90 261608 0.00 0.00 ares_cancel >>> 0.22 314.79 0.73 127417 0.00 0.00 curl_multi_remove_handle >>> 0.19 315.40 0.62 8906301 0.00 0.00 client_tracing_function >>> 0.17 315.97 0.57 135305 0.00 0.00 Curl_num_addresses >>> 0.17 316.54 0.57 181937907 0.00 0.00 addbyter >>> 0.17 317.09 0.55 2374363 0.00 0.00 multi_runsingle >>> 0.16 317.62 0.53 541954 0.00 0.00 ares__init_list_node >>> 0.16 318.14 0.52 494412 0.00 0.00 curl_multi_socket_action >>> 0.14 318.60 0.46 33578897 0.00 0.00 Curl_socket_check >>> 0.13 319.04 0.44 102951052 0.00 0.00 Curl_raw_toupper >>> 0.13 319.48 0.44 787558 0.00 0.00 Curl_readwrite >>> 0.13 319.91 0.44 id_function >>> 0.12 320.29 0.38 391390 0.00 0.00 Curl_hash_str >>> 0.09 320.58 0.29 9216 0.00 0.00 curl_multi_perform >>> 0.08 320.86 0.28 2350908 0.00 0.00 Curl_raw_equal >>> 0.08 321.14 0.28 2019415 0.00 0.00 Curl_splay >>> 0.08 321.40 0.26 520625 0.00 0.00 ossl_connect_common >>> 0.08 321.65 0.25 3532992 0.00 0.00 Curl_pgrsUpdate >>> 0.07 321.87 0.22 131778537 0.00 0.00 curl_strequal >>> 0.05 322.05 0.18 8954118 0.00 0.00 scan_response >>> 0.05 322.23 0.18 533644 0.00 0.00 ares_expand_name >>> 0.05 322.40 0.17 2477345 0.00 0.00 fd_key_compare >>> 0.05 322.57 0.17 2429135 0.00 0.00 Curl_infof >>> 0.05 322.74 0.17 130966 0.00 0.00 singleipconnect >>> 0.05 322.90 0.16 9292 0.00 0.00 curl_multi_socket_all >>> 0.05 323.06 0.16 270137 0.00 0.00 ares__get_hostent >>> 0.05 323.21 0.15 8943657 0.00 0.00 Curl_debug >>> 0.04 323.35 0.14 262493 0.00 0.00 Curl_ssl_getsessionid >>> 0.04 323.48 0.13 566006 0.00 0.00 socket_callback >>> 0.04 323.61 0.13 33983864 0.00 0.00 curlx_tvdiff >>> 0.04 323.74 0.13 132853 0.00 0.00 Curl_http_readwrite_headers >>> 0.04 323.86 0.12 344032 0.00 0.00 epoll_del >>> 0.04 323.98 0.12 223477 0.00 0.00 Curl_poll >>> 0.04 324.10 0.12 8526280 0.00 0.00 Curl_raw_nequal >>> 0.03 324.21 0.11 348839 0.00 0.00 event_del >>> >>> >>> >>> >>> test 2: >>> >>> dns server: dnsmasq@127.0.0.1, 2,000,000 dns entry cache, resolve *.com to one IP address >>> number of domain names in all https URLs: 1 >>> number of client: 1000 >>> CPU usage: 75% >>> SSL TPS: 1300 >>> SSL throughput: 700Mbps >>> >>> # gprof /usr/bin/curl-loader gmon.out -p | head -50 >>> Flat profile: >>> Each sample counts as 0.01 seconds. >>> % cumulative self self total >>> time seconds seconds calls s/call s/call name >>> 8.98 2.45 2.45 5127312 0.00 0.00 dprintf_formatf >>> 8.90 4.88 2.43 1292796503 0.00 0.00 ares__is_list_empty >>> 6.12 6.55 1.67 322461 0.00 0.00 create_conn >>> 4.58 7.80 1.25 419872 0.00 0.00 ares_cancel >>> 4.10 8.92 1.12 2100162 0.00 0.00 Curl_readwrite >>> 3.92 9.99 1.07 4527388 0.00 0.00 Curl_hash_pick >>> 3.43 10.93 0.94 locking_function >>> 3.04 11.76 0.83 909759 0.00 0.00 curl_multi_socket_action >>> 3.00 12.58 0.82 97252 0.00 0.00 Curl_hash_clean_with_criterium >>> 2.71 13.32 0.74 179704016 0.00 0.00 Curl_raw_toupper >>> 2.68 14.05 0.73 4646524 0.00 0.00 multi_runsingle >>> 2.44 14.71 0.67 13862992 0.00 0.00 client_tracing_function >>> 2.42 15.37 0.66 319735 0.00 0.00 curl_multi_remove_handle >>> 2.42 16.03 0.66 22 0.03 0.03 ares__init_list_node >>> 1.94 16.56 0.53 9930244 0.00 0.00 hostcache_timestamp_remove >>> 1.94 17.09 0.53 169320088 0.00 0.00 addbyter >>> 1.80 17.58 0.49 id_function >>> 1.36 17.95 0.37 7057146 0.00 0.00 Curl_pgrsUpdate >>> 1.36 18.32 0.37 16835 0.00 0.00 curl_multi_socket_all >>> 1.25 18.66 0.34 16719 0.00 0.00 curl_multi_perform >>> 1.21 18.99 0.33 329403 0.00 0.00 Curl_http_readwrite_headers >>> 1.17 19.31 0.32 4036624 0.00 0.00 Curl_splay >>> 1.14 19.62 0.31 20838902 0.00 0.00 Curl_raw_nequal >>> 0.82 19.85 0.23 2764123 0.00 0.00 Curl_infof >>> 0.81 20.07 0.22 5933857 0.00 0.00 ossl_recv >>> 0.77 20.28 0.21 323112 0.00 0.00 Curl_splayremovebyaddr >>> 0.75 20.48 0.21 5943991 0.00 0.00 Curl_read >>> 0.70 20.67 0.19 425174 0.00 0.00 event_del >>> 0.70 20.86 0.19 98652 0.00 0.00 Curl_if_is_interface_name >>> 0.70 21.05 0.19 8677733 0.00 0.00 Curl_socket_check >>> 0.59 21.21 0.16 4129554 0.00 0.00 fd_key_compare >>> 0.57 21.37 0.16 14017019 0.00 0.00 Curl_debug >>> 0.55 21.52 0.15 9376861 0.00 0.00 stat_data_in_add >>> 0.55 21.67 0.15 6840468 0.00 0.00 Curl_timeleft >>> 0.55 21.82 0.15 2054602 0.00 0.00 Curl_raw_equal >>> 0.55 21.97 0.15 1291611 0.00 0.00 Curl_expire >>> 0.51 22.11 0.14 16570063 0.00 0.00 curlx_tvnow >>> 0.51 22.25 0.14 13863284 0.00 0.00 scan_response >>> 0.51 22.39 0.14 5498864 0.00 0.00 Curl_setopt >>> 0.48 22.52 0.13 97490291 0.00 0.00 curl_strequal >>> 0.44 22.64 0.12 324253 0.00 0.00 Curl_http >>> 0.44 22.76 0.12 394386 0.00 0.00 ossl_connect_common >>> 0.40 22.87 0.11 3581335 0.00 0.00 Curl_getinfo >>> 0.38 22.97 0.11 26983092 0.00 0.00 alloc_addbyter >>> 0.37 23.07 0.10 8729044 0.00 0.00 Curl_client_write >>> >>> >>> ------------------------------------------------------------------------------ >>> Try before you buy = See our experts in action! >>> The most comprehensive online learning library for Microsoft developers >>> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, >>> Metro Style Apps, more. Free future releases when you subscribe now! >>> http://p.sf.net/sfu/learndevnow-dev2 >>> _______________________________________________ >>> curl-loader-devel mailing list >>> cur...@li... >>> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel >>> >> >> >> >> -- >> Regards, >> Robert Iakobashvili, Ph.D. >> >> Home: http://www.ghotit.com >> Blog: http://www.ghotit.com/category/ghotit-blogs/ >> Twitter: http://twitter.com/ghotit >> Facebook: http://facebook.com/ghotit >> ...................................................................... >> Ghotit Dyslexia >> Assistive technology that understands you >> ...................................................................... >> >> >> ------------------------------------------------------------------------------ >> Try before you buy = See our experts in action! >> The most comprehensive online learning library for Microsoft developers >> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, >> Metro Style Apps, more. Free future releases when you subscribe now! >> http://p.sf.net/sfu/learndevnow-dev2 >> _______________________________________________ >> curl-loader-devel mailing list >> cur...@li... >> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel >> > > > ------------------------------------------------------------------------------ > Try before you buy = See our experts in action! > The most comprehensive online learning library for Microsoft developers > is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, > Metro Style Apps, more. Free future releases when you subscribe now! > http://p.sf.net/sfu/learndevnow-dev2 > _______________________________________________ > curl-loader-devel mailing list > cur...@li... > https://lists.sourceforge.net/lists/listinfo/curl-loader-devel > -- Regards, Robert Iakobashvili, Ph.D. Home: http://www.ghotit.com Blog: http://www.ghotit.com/category/ghotit-blogs/ Twitter: http://twitter.com/ghotit Facebook: http://facebook.com/ghotit ...................................................................... Ghotit Dyslexia Assistive technology that understands you ...................................................................... |
25 messages has been excluded from this view by a project administrator.
