Re: HTTPS throughput too low

[Robert I. <cor...@gm...>]

Hi wei guo,

It's a libcurl issue when built with openssl.

More throughput could be delivered, probably, if to build libcurl
against some another TLS/SSL library.

You can try it.

Regards,
Robert Iakobashvili, Ph.D.

Home:       http://www.ghotit.com
Blog:         http://www.ghotit.com/category/ghotit-blogs/
Twitter:      http://twitter.com/ghotit
Facebook: http://facebook.com/ghotit
......................................................................
                   Ghotit Dyslexia
Assistive technology that understands you
......................................................................


On Mon, Aug 27, 2012 at 5:38 AM, wei guo <wei...@gm...> wrote:
> hi,all:
> do you have any conclusion for this issue？ I ran into the same problem, I'll
> look for answers for this.
>
> On Wed, Oct 12, 2011 at 4:51 PM, Robert Iakobashvili <cor...@gm...>
> wrote:
>>
>> Hi CK Zhang,
>>
>> Now it clear what happens.
>>
>> curl-loader:
>> - establishes TCP connections;
>> - makes expensive TLS/SSL handshake
>> - fetches the file.
>>
>> After fetching file, curl-loader closes the TCP-connection.
>>
>> Next cycle it establishes the TCP-connection and makes the expensive
>> TLS/SSL handshake again,
>> and fetches the file.
>>
>> To achieve high throughput libcurl needs to be configured to re-use
>> the established
>> TCP connection and TLS/SSL for the next file transfer and not to
>> re-establish it every cycle.
>>
>> I suggest you to look in curl (libcurl) options and I will also be doing
>> that.
>>
>> Thanks for your reporting.
>>
>> On Wed, Oct 12, 2011 at 2:59 AM, CK Zhang <ck...@fo...> wrote:
>> >
>> > Attached logs, and output from console
>> >
>> > ============  loading batch is: Apach
>> > ======================================
>> >
>> > ----------------------------------------------------------------------------
>> > ----
>> > Interval stats (latest:3 sec, clients:100, CAPS-curr:60):
>> > H/F
>> >
>> > Req:0,1xx:0,2xx:0,3xx:0,4xx:0,5xx:0,Err:0,T-Err:0,D:0ms,D-2xx:0ms,Ti:0B/s,To
>> > :0B/s
>> > H/F/S
>> >
>> > Req:200,1xx:0,2xx:201,3xx:0,4xx:0,5xx:0,Err:0,T-Err:0,D:553ms,D-2xx:553ms,Ti
>> > :6854220B/s,To:18800B/s
>> >
>> > ----------------------------------------------------------------------------
>> > ----
>> > Summary stats (runs:48 secs, CAPS-average:60):
>> > H/F
>> >
>> > Req:0,1xx:0,2xx:0,3xx:0,4xx:0,5xx:0,Err:0,T-Err:0,D:0ms,D-2xx:0ms,Ti:0B/s,To
>> > :0B/s
>> > H/F/S
>> >
>> > Req:2878,1xx:0,2xx:2873,3xx:0,4xx:0,5xx:0,Err:0,T-Err:0,D:614ms,D-2xx:614ms,
>> > Ti:6087722B/s,To:17182B/s
>> >
>> > ============================================================================
>> > =====
>> >  Manual: clients:max[100],curr[100]. Inc num: [+|*].
>> >
>> > ============================================================================
>> > =====
>> >
>> > Best Regards,
>> > CK
>> > #8042
>> >
>> >
>> > -----Original Message-----
>> > From: Robert Iakobashvili [mailto:cor...@gm...]
>> > Sent: Tuesday, October 11, 2011 5:49 PM
>> > To: curl-loader-devel
>> > Subject: Re: HTTPS throughput too low
>> >
>> > Please, try
>> >
>> > CLIENTS_NUM_MAX=100
>> > CLIENTS_NUM_START=100
>> >
>> > and comment out
>> > #CLIENTS_RAMPUP_INC=10
>> >
>> > Say, what happens with traffic and send me the log.
>> >
>> > On Tue, Oct 11, 2011 at 11:36 AM, CK Zhang <ck...@fo...> wrote:
>> >> Attached :) thanks!
>> >>
>> >> Best Regards,
>> >> CK
>> >> #8042
>> >>
>> >>
>> >> -----Original Message-----
>> >> From: Robert Iakobashvili [mailto:cor...@gm...]
>> >> Sent: Tuesday, October 11, 2011 5:33 PM
>> >> To: curl-loader-devel
>> >> Subject: Re: HTTPS throughput too low
>> >>
>> >> Could you send me it zipped not rarred?
>> >> Thanks!
>> >>
>> >> On Tue, Oct 11, 2011 at 11:30 AM, CK Zhang <ck...@fo...>
>> >> wrote:
>> >>>  Dear Robert
>> >>>
>> >>> Attached test log by command "./curl-loader -f https.conf -v"
>> >>>
>> >>> Best Regards,
>> >>> CK
>> >>> #8042
>> >>>
>> >>> -----Original Message-----
>> >>> From: Robert Iakobashvili [mailto:cor...@gm...]
>> >>> Sent: Tuesday, October 11, 2011 5:19 PM
>> >>> To: curl-loader-devel
>> >>> Subject: Re: HTTPS throughput too low
>> >>>
>> >>> Dear CK Zhang,
>> >>>
>> >>> Can you run HTTPS sessions with -v detailed logging and say if
>> >>> you seen in logs something interesting.
>> >>> Thanks,
>> >>> Robert
>> >>>
>> >>>
>> >>>
>> >>> On Tue, Oct 11, 2011 at 11:14 AM, CK Zhang <ck...@fo...>
>> >>> wrote:
>> >>>> Dear Robert
>> >>>>
>> >>>>
>> >>>>
>> >>>> Thanks a lot for your replay,
>> >>>>
>> >>>>
>> >>>>
>> >>>> Here are some answer about your questions
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>> What happens if you try HTTP? Does it impact throughput?
>> >>>>
>> >>>>  It is normal and perfect if I test with HTTP, The throughput can
>> >>>> reach
>> >> to
>> >>>> 950Mbps.
>> >>>>
>> >>>>
>> >>>>
>> >>>> Which tools you are using to measure throughput?
>> >>>>
>> >>>> There are 2 ways I used,
>> >>>>
>> >>>>  1 check the log summary form cur-loader like this”
>> >>>> Ti:5261372B/s,To:18375B/s”
>> >>>>
>> >>>> 2 check real time traffic speed on switch with connected test PC and
>> > HTTP
>> >>>> server
>> >>>>
>> >>>>
>> >>>>
>> >>>> What happens, if you try just to ftp some large file between the
>> >> machines?
>> >>>>
>> >>>> NO test
>> >>>>
>> >>>> What about CPU usage at both client and server?
>> >>>>
>> >>>> Both they are busy, about 80%
>> >>>>
>> >>>> Which HTTP server is in use?
>> >>>>
>> >>>>  Apache/2.2.21
>> >>>>
>> >>>> Have you looked into logs of curl-loader and your web-server?
>> >>>>
>> >>>> Checked log, not find abnormal issue.
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>> Today, I used Loadrunner11 to simulate https traffic with same
>> >>> environments
>> >>>> (same apache server) , it can reach to 110Mpbs. But still only get
>> >>>  800kbps
>> >>>> with cur-loader.
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>> Best Regards,
>> >>>>
>> >>>> CK
>> >>>>
>> >>>> #8042
>> >>>>
>> >>>>
>> >>>>
>> >>>> From: Robert Iakobashvili [mailto:cor...@gm...]
>> >>>> Sent: Tuesday, September 27, 2011 1:19 PM
>> >>>> To: curl-loader-devel
>> >>>> Subject: Re: HTTPS throughput too low
>> >>>>
>> >>>>
>> >>>>
>> >>>> Dear CK Zhang,
>> >>>>
>> >>>> On Mon, Sep 26, 2011 at 5:01 AM, CK Zhang <ck...@fo...>
>> >>>> wrote:
>> >>>>
>> >>>>
>> >>>>
>> >>>> I want to get high https stream with curl loader, but the Https
>> >> throughput
>> >>>> is about 800kbps between two PC, Http throughput can reach 950Mbps in
>> >> that
>> >>> 2
>> >>>> pcs, I do know if some have experience like this, and get a high
>> >>>> https
>> >>>> throughput with curl-loader, Thanks for any supports and help!!
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>> Here is my configuration:
>> >>>>
>> >>>>
>> >>>>
>> >>>> PC1:CPU Inter E2200, Mem 2G, 1000M NIC
>> >>>>
>> >>>> OpenSSL:
>> >>>>
>> >>>> openssl-devel-1.0.0-4.el6_0.2.x86_64
>> >>>>
>> >>>> openssl-1.0.0-4.el6_0.2.x86_64
>> >>>>
>> >>>> openssl-perl-1.0.0-4.el6_0.2.x86_64
>> >>>>
>> >>>>
>> >>>>
>> >>>> curl-loader-0.53
>> >>>>
>> >>>>                                ########### GENERAL SECTION
>> >>>> ################################
>> >>>>
>> >>>> BATCH_NAME= Apach
>> >>>>
>> >>>> CLIENTS_NUM_MAX=400
>> >>>>
>> >>>> CLIENTS_NUM_START=100
>> >>>>
>> >>>> CLIENTS_RAMPUP_INC=10
>> >>>>
>> >>>> INTERFACE   =eth2
>> >>>>
>> >>>> NETMASK=16
>> >>>>
>> >>>> IP_ADDR_MIN= 172.16.76.150
>> >>>>
>> >>>> IP_ADDR_MAX= 172.16.246.250  #Actually - this is for self-control
>> >>>>
>> >>>> CYCLES_NUM= -1
>> >>>>
>> >>>> URLS_NUM= 1
>> >>>>
>> >>>>
>> >>>>
>> >>>> ########### URL SECTION ####################################
>> >>>>
>> >>>>
>> >>>>
>> >>>> URL=https://172.16.76.142/was/100KB.txt # the size of 100KB.txt is
>> >>>> 100KB
>> >>>>
>> >>>> #URL=http://localhost/ACE-INSTALL.html
>> >>>>
>> >>>> URL_SHORT_NAME="local-index"
>> >>>>
>> >>>> REQUEST_TYPE=GET
>> >>>>
>> >>>> TIMER_URL_COMPLETION = 0      # In msec. When positive, Now it is
>> >> enforced
>> >>>> by cancelling url fetch on timeout
>> >>>>
>> >>>> TIMER_AFTER_URL_SLEEP =1000
>> >>>>
>> >>>> ~
>> >>>>
>> >>>>
>> >>>>
>> >>>> PC2. Works as Webserver,CPU Intel E7600,Mem 4G, 1000M NIC
>> >>>>
>> >>>> Server version: Apache/2.2.21 (Unix)
>> >>>>
>> >>>> Server built:   Sep 21 2011 16:19:43
>> >>>>
>> >>>> openssl-devel-1.0.0-4.el6_0.2.x86_64
>> >>>>
>> >>>> openssl-1.0.0-4.el6_0.2.x86_64
>> >>>>
>> >>>> openssl-perl-1.0.0-4.el6_0.2.x86_64
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>> Interesting.
>> >>>>
>> >>>>
>> >>>>
>> >>>> What happens if you try HTTP? Does it impact throughput?
>> >>>>
>> >>>>
>> >>>>
>> >>>> Which tools you are using to measure throughput?
>> >>>>
>> >>>>
>> >>>>
>> >>>> What happens, if you try just to ftp some large file between the
>> >> machines?
>> >>>>
>> >>>>
>> >>>>
>> >>>> What about CPU usage at both client and server?
>> >>>>
>> >>>> Which HTTP server is in use?
>> >>>>
>> >>>>
>> >>>>
>> >>>> Have you looked into logs of curl-loader and your web-server?
>> >>>>
>> >>>>
>> >>>>
>> >>>> --
>> >>>> Regards,
>> >>>> Robert Iakobashvili, Ph.D.
>> >>>>
>> >>>>
>> >>>>
>> >>>> Home:       http://www.ghotit.com
>> >>>>
>> >>>> Blog:         http://dyslexia-blog.ghotit.com
>> >>>>
>> >>>> Twitter:      http://twitter.com/ghotit
>> >>>>
>> >>>> Facebook: http://facebook.com/ghotit
>> >>>>
>> >>>>
>> >>>> ......................................................................
>> >>>>
>> >>>>                    Ghotit Dyslexia
>> >>>>
>> >>>> Assistive technology that understands you
>> >>>>
>> >>>>
>> >>>> ......................................................................
>> >>>>
>> >>>>
>> >>>>
>> >>>> ________________________________
>> >>>>
>> >>>> ***  Please note that this message and any attachments may contain
>> >>>> confidential
>> >>>> and proprietary material and information and are intended only for
>> >>>> the
>> >> use
>> >>>> of
>> >>>> the intended recipient(s). If you are not the intended recipient, you
>> > are
>> >>>> hereby
>> >>>> notified that any review, use, disclosure, dissemination,
>> >>>> distribution
>> > or
>> >>>> copying
>> >>>> of this message and any attachments is strictly prohibited. If you
>> >>>> have
>> >>>> received
>> >>>> this email in error, please immediately notify the sender and destroy
>> >> this
>> >>>> e-mail
>> >>>> and any attachments and all copies, whether electronic or printed.
>> >>>> Please also note that any views, opinions, conclusions or commitments
>> >>>> expressed
>> >>>> in this message are those of the individual sender and do not
>> > necessarily
>> >>>> reflect
>> >>>> the views of Fortinet, Inc., its affiliates, and emails are not
>> >>>> binding
>> >> on
>> >>>> Fortinet and only a writing manually signed by Fortinet's General
>> > Counsel
>> >>>> can be
>> >>>> a binding commitment of Fortinet to Fortinet's customers or partners.
>> >>> Thank
>> >>>> you. ***
>> >>>>
>> >>>> ________________________________
>> >>>>
>> >>>>
>> >>>
>> >>
>> >
>> > ----------------------------------------------------------------------------
>> >>> --
>> >>>> All the data continuously generated in your IT infrastructure
>> >>>> contains a
>> >>>> definitive record of customers, application performance, security
>> >>>> threats, fraudulent activity and more. Splunk takes this data and
>> >>>> makes
>> >>>> sense of it. Business sense. IT sense. Common sense.
>> >>>> http://p.sf.net/sfu/splunk-d2d-oct
>> >>>> _______________________________________________
>> >>>> curl-loader-devel mailing list
>> >>>> cur...@li...
>> >>>> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel
>> >>>>
>> >>>>
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> Regards,
>> >>> Robert Iakobashvili, Ph.D.
>> >>>
>> >>> Home:       http://www.ghotit.com
>> >>> Blog:         http://dyslexia-blog.ghotit.com
>> >>> Twitter:      http://twitter.com/ghotit
>> >>> Facebook: http://facebook.com/ghotit
>> >>> ......................................................................
>> >>>                    Ghotit Dyslexia
>> >>> Assistive technology that understands you
>> >>> ......................................................................
>> >>>
>> >>>
>> >>
>> >
>> > ----------------------------------------------------------------------------
>> >>> --
>> >>> All the data continuously generated in your IT infrastructure contains
>> >>> a
>> >>> definitive record of customers, application performance, security
>> >>> threats, fraudulent activity and more. Splunk takes this data and
>> >>> makes
>> >>> sense of it. Business sense. IT sense. Common sense.
>> >>> http://p.sf.net/sfu/splunk-d2d-oct
>> >>> _______________________________________________
>> >>> curl-loader-devel mailing list
>> >>> cur...@li...
>> >>> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel
>> >>>
>> >>>
>> >>> ***  Please note that this message and any attachments may contain
>> >> confidential
>> >>> and proprietary material and information and are intended only for the
>> > use
>> >> of
>> >>> the intended recipient(s). If you are not the intended recipient, you
>> >>> are
>> >> hereby
>> >>> notified that any review, use, disclosure, dissemination, distribution
>> >>> or
>> >> copying
>> >>> of this message and any attachments is strictly prohibited. If you
>> >>> have
>> >> received
>> >>> this email in error, please immediately notify the sender and destroy
>> > this
>> >> e-mail
>> >>> and any attachments and all copies, whether electronic or printed.
>> >>> Please also note that any views, opinions, conclusions or commitments
>> >> expressed
>> >>> in this message are those of the individual sender and do not
>> >>> necessarily
>> >> reflect
>> >>> the views of Fortinet, Inc., its affiliates, and emails are not
>> >>> binding
>> > on
>> >>> Fortinet and only a writing manually signed by Fortinet's General
>> >>> Counsel
>> >> can be
>> >>> a binding commitment of Fortinet to Fortinet's customers or partners.
>> >> Thank you. ***
>> >>>
>> >>>
>> >>
>> >
>> > ----------------------------------------------------------------------------
>> >> --
>> >>> All the data continuously generated in your IT infrastructure contains
>> >>> a
>> >>> definitive record of customers, application performance, security
>> >>> threats, fraudulent activity and more. Splunk takes this data and
>> >>> makes
>> >>> sense of it. Business sense. IT sense. Common sense.
>> >>> http://p.sf.net/sfu/splunk-d2d-oct
>> >>> _______________________________________________
>> >>> curl-loader-devel mailing list
>> >>> cur...@li...
>> >>> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel
>> >>>
>> >>>
>> >>
>> >>
>> >>
>> >> --
>> >> Regards,
>> >> Robert Iakobashvili, Ph.D.
>> >>
>> >> Home:       http://www.ghotit.com
>> >> Blog:         http://dyslexia-blog.ghotit.com
>> >> Twitter:      http://twitter.com/ghotit
>> >> Facebook: http://facebook.com/ghotit
>> >> ......................................................................
>> >>                    Ghotit Dyslexia
>> >> Assistive technology that understands you
>> >> ......................................................................
>> >>
>> >>
>> >
>> > ----------------------------------------------------------------------------
>> >> --
>> >> All the data continuously generated in your IT infrastructure contains
>> >> a
>> >> definitive record of customers, application performance, security
>> >> threats, fraudulent activity and more. Splunk takes this data and makes
>> >> sense of it. Business sense. IT sense. Common sense.
>> >> http://p.sf.net/sfu/splunk-d2d-oct
>> >> _______________________________________________
>> >> curl-loader-devel mailing list
>> >> cur...@li...
>> >> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel
>> >>
>> >>
>> >> ***  Please note that this message and any attachments may contain
>> > confidential
>> >> and proprietary material and information and are intended only for the
>> >> use
>> > of
>> >> the intended recipient(s). If you are not the intended recipient, you
>> >> are
>> > hereby
>> >> notified that any review, use, disclosure, dissemination, distribution
>> >> or
>> > copying
>> >> of this message and any attachments is strictly prohibited. If you have
>> > received
>> >> this email in error, please immediately notify the sender and destroy
>> >> this
>> > e-mail
>> >> and any attachments and all copies, whether electronic or printed.
>> >> Please also note that any views, opinions, conclusions or commitments
>> > expressed
>> >> in this message are those of the individual sender and do not
>> >> necessarily
>> > reflect
>> >> the views of Fortinet, Inc., its affiliates, and emails are not binding
>> >> on
>> >> Fortinet and only a writing manually signed by Fortinet's General
>> >> Counsel
>> > can be
>> >> a binding commitment of Fortinet to Fortinet's customers or partners.
>> > Thank you. ***
>> >>
>> >>
>> >
>> > ----------------------------------------------------------------------------
>> > --
>> >> All the data continuously generated in your IT infrastructure contains
>> >> a
>> >> definitive record of customers, application performance, security
>> >> threats, fraudulent activity and more. Splunk takes this data and makes
>> >> sense of it. Business sense. IT sense. Common sense.
>> >> http://p.sf.net/sfu/splunk-d2d-oct
>> >> _______________________________________________
>> >> curl-loader-devel mailing list
>> >> cur...@li...
>> >> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel
>> >>
>> >>
>> >
>> >
>> >
>> > --
>> > Regards,
>> > Robert Iakobashvili, Ph.D.
>> >
>> > Home:       http://www.ghotit.com
>> > Blog:         http://dyslexia-blog.ghotit.com
>> > Twitter:      http://twitter.com/ghotit
>> > Facebook: http://facebook.com/ghotit
>> > ......................................................................
>> >                    Ghotit Dyslexia
>> > Assistive technology that understands you
>> > ......................................................................
>> >
>> >
>> > ----------------------------------------------------------------------------
>> > --
>> > All the data continuously generated in your IT infrastructure contains a
>> > definitive record of customers, application performance, security
>> > threats, fraudulent activity and more. Splunk takes this data and makes
>> > sense of it. Business sense. IT sense. Common sense.
>> > http://p.sf.net/sfu/splunk-d2d-oct
>> > _______________________________________________
>> > curl-loader-devel mailing list
>> > cur...@li...
>> > https://lists.sourceforge.net/lists/listinfo/curl-loader-devel
>> >
>> >
>> > ***  Please note that this message and any attachments may contain
>> > confidential
>> > and proprietary material and information and are intended only for the
>> > use of
>> > the intended recipient(s). If you are not the intended recipient, you
>> > are hereby
>> > notified that any review, use, disclosure, dissemination, distribution
>> > or copying
>> > of this message and any attachments is strictly prohibited. If you have
>> > received
>> > this email in error, please immediately notify the sender and destroy
>> > this e-mail
>> > and any attachments and all copies, whether electronic or printed.
>> > Please also note that any views, opinions, conclusions or commitments
>> > expressed
>> > in this message are those of the individual sender and do not
>> > necessarily reflect
>> > the views of Fortinet, Inc., its affiliates, and emails are not binding
>> > on
>> > Fortinet and only a writing manually signed by Fortinet's General
>> > Counsel can be
>> > a binding commitment of Fortinet to Fortinet's customers or partners.
>> > Thank you. ***
>> >
>> >
>> > ------------------------------------------------------------------------------
>> > All the data continuously generated in your IT infrastructure contains a
>> > definitive record of customers, application performance, security
>> > threats, fraudulent activity and more. Splunk takes this data and makes
>> > sense of it. Business sense. IT sense. Common sense.
>> > http://p.sf.net/sfu/splunk-d2d-oct
>> > _______________________________________________
>> > curl-loader-devel mailing list
>> > cur...@li...
>> > https://lists.sourceforge.net/lists/listinfo/curl-loader-devel
>> >
>> >
>>
>>
>>
>> --
>> Regards,
>> Robert Iakobashvili, Ph.D.
>>
>> Home:       http://www.ghotit.com
>> Blog:         http://dyslexia-blog.ghotit.com
>> Twitter:      http://twitter.com/ghotit
>> Facebook: http://facebook.com/ghotit
>> ......................................................................
>>                    Ghotit Dyslexia
>> Assistive technology that understands you
>> ......................................................................
>>
>>
>> ------------------------------------------------------------------------------
>> All the data continuously generated in your IT infrastructure contains a
>> definitive record of customers, application performance, security
>> threats, fraudulent activity and more. Splunk takes this data and makes
>> sense of it. Business sense. IT sense. Common sense.
>> http://p.sf.net/sfu/splunk-d2d-oct
>> _______________________________________________
>> curl-loader-devel mailing list
>> cur...@li...
>> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel
>
>
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> curl-loader-devel mailing list
> cur...@li...
> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel
>