Re: HTTPS throughput too low

[Robert I. <cor...@gm...>]

Hi CK Zhang,

Now it clear what happens.

curl-loader:
- establishes TCP connections;
- makes expensive TLS/SSL handshake
- fetches the file.

After fetching file, curl-loader closes the TCP-connection.

Next cycle it establishes the TCP-connection and makes the expensive
TLS/SSL handshake again,
and fetches the file.

To achieve high throughput libcurl needs to be configured to re-use
the established
TCP connection and TLS/SSL for the next file transfer and not to
re-establish it every cycle.

I suggest you to look in curl (libcurl) options and I will also be doing that.

Thanks for your reporting.

On Wed, Oct 12, 2011 at 2:59 AM, CK Zhang <ck...@fo...> wrote:
>
> Attached logs, and output from console
>
> ============  loading batch is: Apach
> ======================================
> ----------------------------------------------------------------------------
> ----
> Interval stats (latest:3 sec, clients:100, CAPS-curr:60):
> H/F
> Req:0,1xx:0,2xx:0,3xx:0,4xx:0,5xx:0,Err:0,T-Err:0,D:0ms,D-2xx:0ms,Ti:0B/s,To
> :0B/s
> H/F/S
> Req:200,1xx:0,2xx:201,3xx:0,4xx:0,5xx:0,Err:0,T-Err:0,D:553ms,D-2xx:553ms,Ti
> :6854220B/s,To:18800B/s
> ----------------------------------------------------------------------------
> ----
> Summary stats (runs:48 secs, CAPS-average:60):
> H/F
> Req:0,1xx:0,2xx:0,3xx:0,4xx:0,5xx:0,Err:0,T-Err:0,D:0ms,D-2xx:0ms,Ti:0B/s,To
> :0B/s
> H/F/S
> Req:2878,1xx:0,2xx:2873,3xx:0,4xx:0,5xx:0,Err:0,T-Err:0,D:614ms,D-2xx:614ms,
> Ti:6087722B/s,To:17182B/s
> ============================================================================
> =====
>  Manual: clients:max[100],curr[100]. Inc num: [+|*].
> ============================================================================
> =====
>
> Best Regards,
> CK
> #8042
>
>
> -----Original Message-----
> From: Robert Iakobashvili [mailto:cor...@gm...]
> Sent: Tuesday, October 11, 2011 5:49 PM
> To: curl-loader-devel
> Subject: Re: HTTPS throughput too low
>
> Please, try
>
> CLIENTS_NUM_MAX=100
> CLIENTS_NUM_START=100
>
> and comment out
> #CLIENTS_RAMPUP_INC=10
>
> Say, what happens with traffic and send me the log.
>
> On Tue, Oct 11, 2011 at 11:36 AM, CK Zhang <ck...@fo...> wrote:
>> Attached :) thanks!
>>
>> Best Regards,
>> CK
>> #8042
>>
>>
>> -----Original Message-----
>> From: Robert Iakobashvili [mailto:cor...@gm...]
>> Sent: Tuesday, October 11, 2011 5:33 PM
>> To: curl-loader-devel
>> Subject: Re: HTTPS throughput too low
>>
>> Could you send me it zipped not rarred?
>> Thanks!
>>
>> On Tue, Oct 11, 2011 at 11:30 AM, CK Zhang <ck...@fo...> wrote:
>>>  Dear Robert
>>>
>>> Attached test log by command "./curl-loader -f https.conf -v"
>>>
>>> Best Regards,
>>> CK
>>> #8042
>>>
>>> -----Original Message-----
>>> From: Robert Iakobashvili [mailto:cor...@gm...]
>>> Sent: Tuesday, October 11, 2011 5:19 PM
>>> To: curl-loader-devel
>>> Subject: Re: HTTPS throughput too low
>>>
>>> Dear CK Zhang,
>>>
>>> Can you run HTTPS sessions with -v detailed logging and say if
>>> you seen in logs something interesting.
>>> Thanks,
>>> Robert
>>>
>>>
>>>
>>> On Tue, Oct 11, 2011 at 11:14 AM, CK Zhang <ck...@fo...> wrote:
>>>> Dear Robert
>>>>
>>>>
>>>>
>>>> Thanks a lot for your replay,
>>>>
>>>>
>>>>
>>>> Here are some answer about your questions
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> What happens if you try HTTP? Does it impact throughput?
>>>>
>>>>  It is normal and perfect if I test with HTTP, The throughput can reach
>> to
>>>> 950Mbps.
>>>>
>>>>
>>>>
>>>> Which tools you are using to measure throughput?
>>>>
>>>> There are 2 ways I used,
>>>>
>>>>  1 check the log summary form cur-loader like this”
>>>> Ti:5261372B/s,To:18375B/s”
>>>>
>>>> 2 check real time traffic speed on switch with connected test PC and
> HTTP
>>>> server
>>>>
>>>>
>>>>
>>>> What happens, if you try just to ftp some large file between the
>> machines?
>>>>
>>>> NO test
>>>>
>>>> What about CPU usage at both client and server?
>>>>
>>>> Both they are busy, about 80%
>>>>
>>>> Which HTTP server is in use?
>>>>
>>>>  Apache/2.2.21
>>>>
>>>> Have you looked into logs of curl-loader and your web-server?
>>>>
>>>> Checked log, not find abnormal issue.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Today, I used Loadrunner11 to simulate https traffic with same
>>> environments
>>>> (same apache server) , it can reach to 110Mpbs. But still only get
>>>  800kbps
>>>> with cur-loader.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Best Regards,
>>>>
>>>> CK
>>>>
>>>> #8042
>>>>
>>>>
>>>>
>>>> From: Robert Iakobashvili [mailto:cor...@gm...]
>>>> Sent: Tuesday, September 27, 2011 1:19 PM
>>>> To: curl-loader-devel
>>>> Subject: Re: HTTPS throughput too low
>>>>
>>>>
>>>>
>>>> Dear CK Zhang,
>>>>
>>>> On Mon, Sep 26, 2011 at 5:01 AM, CK Zhang <ck...@fo...> wrote:
>>>>
>>>>
>>>>
>>>> I want to get high https stream with curl loader, but the Https
>> throughput
>>>> is about 800kbps between two PC, Http throughput can reach 950Mbps in
>> that
>>> 2
>>>> pcs, I do know if some have experience like this, and get a high https
>>>> throughput with curl-loader, Thanks for any supports and help!!
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Here is my configuration:
>>>>
>>>>
>>>>
>>>> PC1:CPU Inter E2200, Mem 2G, 1000M NIC
>>>>
>>>> OpenSSL:
>>>>
>>>> openssl-devel-1.0.0-4.el6_0.2.x86_64
>>>>
>>>> openssl-1.0.0-4.el6_0.2.x86_64
>>>>
>>>> openssl-perl-1.0.0-4.el6_0.2.x86_64
>>>>
>>>>
>>>>
>>>> curl-loader-0.53
>>>>
>>>>                                ########### GENERAL SECTION
>>>> ################################
>>>>
>>>> BATCH_NAME= Apach
>>>>
>>>> CLIENTS_NUM_MAX=400
>>>>
>>>> CLIENTS_NUM_START=100
>>>>
>>>> CLIENTS_RAMPUP_INC=10
>>>>
>>>> INTERFACE   =eth2
>>>>
>>>> NETMASK=16
>>>>
>>>> IP_ADDR_MIN= 172.16.76.150
>>>>
>>>> IP_ADDR_MAX= 172.16.246.250  #Actually - this is for self-control
>>>>
>>>> CYCLES_NUM= -1
>>>>
>>>> URLS_NUM= 1
>>>>
>>>>
>>>>
>>>> ########### URL SECTION ####################################
>>>>
>>>>
>>>>
>>>> URL=https://172.16.76.142/was/100KB.txt # the size of 100KB.txt is 100KB
>>>>
>>>> #URL=http://localhost/ACE-INSTALL.html
>>>>
>>>> URL_SHORT_NAME="local-index"
>>>>
>>>> REQUEST_TYPE=GET
>>>>
>>>> TIMER_URL_COMPLETION = 0      # In msec. When positive, Now it is
>> enforced
>>>> by cancelling url fetch on timeout
>>>>
>>>> TIMER_AFTER_URL_SLEEP =1000
>>>>
>>>> ~
>>>>
>>>>
>>>>
>>>> PC2. Works as Webserver,CPU Intel E7600,Mem 4G, 1000M NIC
>>>>
>>>> Server version: Apache/2.2.21 (Unix)
>>>>
>>>> Server built:   Sep 21 2011 16:19:43
>>>>
>>>> openssl-devel-1.0.0-4.el6_0.2.x86_64
>>>>
>>>> openssl-1.0.0-4.el6_0.2.x86_64
>>>>
>>>> openssl-perl-1.0.0-4.el6_0.2.x86_64
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Interesting.
>>>>
>>>>
>>>>
>>>> What happens if you try HTTP? Does it impact throughput?
>>>>
>>>>
>>>>
>>>> Which tools you are using to measure throughput?
>>>>
>>>>
>>>>
>>>> What happens, if you try just to ftp some large file between the
>> machines?
>>>>
>>>>
>>>>
>>>> What about CPU usage at both client and server?
>>>>
>>>> Which HTTP server is in use?
>>>>
>>>>
>>>>
>>>> Have you looked into logs of curl-loader and your web-server?
>>>>
>>>>
>>>>
>>>> --
>>>> Regards,
>>>> Robert Iakobashvili, Ph.D.
>>>>
>>>>
>>>>
>>>> Home:       http://www.ghotit.com
>>>>
>>>> Blog:         http://dyslexia-blog.ghotit.com
>>>>
>>>> Twitter:      http://twitter.com/ghotit
>>>>
>>>> Facebook: http://facebook.com/ghotit
>>>>
>>>> ......................................................................
>>>>
>>>>                    Ghotit Dyslexia
>>>>
>>>> Assistive technology that understands you
>>>>
>>>> ......................................................................
>>>>
>>>>
>>>>
>>>> ________________________________
>>>>
>>>> ***  Please note that this message and any attachments may contain
>>>> confidential
>>>> and proprietary material and information and are intended only for the
>> use
>>>> of
>>>> the intended recipient(s). If you are not the intended recipient, you
> are
>>>> hereby
>>>> notified that any review, use, disclosure, dissemination, distribution
> or
>>>> copying
>>>> of this message and any attachments is strictly prohibited. If you have
>>>> received
>>>> this email in error, please immediately notify the sender and destroy
>> this
>>>> e-mail
>>>> and any attachments and all copies, whether electronic or printed.
>>>> Please also note that any views, opinions, conclusions or commitments
>>>> expressed
>>>> in this message are those of the individual sender and do not
> necessarily
>>>> reflect
>>>> the views of Fortinet, Inc., its affiliates, and emails are not binding
>> on
>>>> Fortinet and only a writing manually signed by Fortinet's General
> Counsel
>>>> can be
>>>> a binding commitment of Fortinet to Fortinet's customers or partners.
>>> Thank
>>>> you. ***
>>>>
>>>> ________________________________
>>>>
>>>>
>>>
>>
> ----------------------------------------------------------------------------
>>> --
>>>> All the data continuously generated in your IT infrastructure contains a
>>>> definitive record of customers, application performance, security
>>>> threats, fraudulent activity and more. Splunk takes this data and makes
>>>> sense of it. Business sense. IT sense. Common sense.
>>>> http://p.sf.net/sfu/splunk-d2d-oct
>>>> _______________________________________________
>>>> curl-loader-devel mailing list
>>>> cur...@li...
>>>> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Regards,
>>> Robert Iakobashvili, Ph.D.
>>>
>>> Home:       http://www.ghotit.com
>>> Blog:         http://dyslexia-blog.ghotit.com
>>> Twitter:      http://twitter.com/ghotit
>>> Facebook: http://facebook.com/ghotit
>>> ......................................................................
>>>                    Ghotit Dyslexia
>>> Assistive technology that understands you
>>> ......................................................................
>>>
>>>
>>
> ----------------------------------------------------------------------------
>>> --
>>> All the data continuously generated in your IT infrastructure contains a
>>> definitive record of customers, application performance, security
>>> threats, fraudulent activity and more. Splunk takes this data and makes
>>> sense of it. Business sense. IT sense. Common sense.
>>> http://p.sf.net/sfu/splunk-d2d-oct
>>> _______________________________________________
>>> curl-loader-devel mailing list
>>> cur...@li...
>>> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel
>>>
>>>
>>> ***  Please note that this message and any attachments may contain
>> confidential
>>> and proprietary material and information and are intended only for the
> use
>> of
>>> the intended recipient(s). If you are not the intended recipient, you are
>> hereby
>>> notified that any review, use, disclosure, dissemination, distribution or
>> copying
>>> of this message and any attachments is strictly prohibited. If you have
>> received
>>> this email in error, please immediately notify the sender and destroy
> this
>> e-mail
>>> and any attachments and all copies, whether electronic or printed.
>>> Please also note that any views, opinions, conclusions or commitments
>> expressed
>>> in this message are those of the individual sender and do not necessarily
>> reflect
>>> the views of Fortinet, Inc., its affiliates, and emails are not binding
> on
>>> Fortinet and only a writing manually signed by Fortinet's General Counsel
>> can be
>>> a binding commitment of Fortinet to Fortinet's customers or partners.
>> Thank you. ***
>>>
>>>
>>
> ----------------------------------------------------------------------------
>> --
>>> All the data continuously generated in your IT infrastructure contains a
>>> definitive record of customers, application performance, security
>>> threats, fraudulent activity and more. Splunk takes this data and makes
>>> sense of it. Business sense. IT sense. Common sense.
>>> http://p.sf.net/sfu/splunk-d2d-oct
>>> _______________________________________________
>>> curl-loader-devel mailing list
>>> cur...@li...
>>> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel
>>>
>>>
>>
>>
>>
>> --
>> Regards,
>> Robert Iakobashvili, Ph.D.
>>
>> Home:       http://www.ghotit.com
>> Blog:         http://dyslexia-blog.ghotit.com
>> Twitter:      http://twitter.com/ghotit
>> Facebook: http://facebook.com/ghotit
>> ......................................................................
>>                    Ghotit Dyslexia
>> Assistive technology that understands you
>> ......................................................................
>>
>>
> ----------------------------------------------------------------------------
>> --
>> All the data continuously generated in your IT infrastructure contains a
>> definitive record of customers, application performance, security
>> threats, fraudulent activity and more. Splunk takes this data and makes
>> sense of it. Business sense. IT sense. Common sense.
>> http://p.sf.net/sfu/splunk-d2d-oct
>> _______________________________________________
>> curl-loader-devel mailing list
>> cur...@li...
>> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel
>>
>>
>> ***  Please note that this message and any attachments may contain
> confidential
>> and proprietary material and information and are intended only for the use
> of
>> the intended recipient(s). If you are not the intended recipient, you are
> hereby
>> notified that any review, use, disclosure, dissemination, distribution or
> copying
>> of this message and any attachments is strictly prohibited. If you have
> received
>> this email in error, please immediately notify the sender and destroy this
> e-mail
>> and any attachments and all copies, whether electronic or printed.
>> Please also note that any views, opinions, conclusions or commitments
> expressed
>> in this message are those of the individual sender and do not necessarily
> reflect
>> the views of Fortinet, Inc., its affiliates, and emails are not binding on
>> Fortinet and only a writing manually signed by Fortinet's General Counsel
> can be
>> a binding commitment of Fortinet to Fortinet's customers or partners.
> Thank you. ***
>>
>>
> ----------------------------------------------------------------------------
> --
>> All the data continuously generated in your IT infrastructure contains a
>> definitive record of customers, application performance, security
>> threats, fraudulent activity and more. Splunk takes this data and makes
>> sense of it. Business sense. IT sense. Common sense.
>> http://p.sf.net/sfu/splunk-d2d-oct
>> _______________________________________________
>> curl-loader-devel mailing list
>> cur...@li...
>> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel
>>
>>
>
>
>
> --
> Regards,
> Robert Iakobashvili, Ph.D.
>
> Home:       http://www.ghotit.com
> Blog:         http://dyslexia-blog.ghotit.com
> Twitter:      http://twitter.com/ghotit
> Facebook: http://facebook.com/ghotit
> ......................................................................
>                    Ghotit Dyslexia
> Assistive technology that understands you
> ......................................................................
>
> ----------------------------------------------------------------------------
> --
> All the data continuously generated in your IT infrastructure contains a
> definitive record of customers, application performance, security
> threats, fraudulent activity and more. Splunk takes this data and makes
> sense of it. Business sense. IT sense. Common sense.
> http://p.sf.net/sfu/splunk-d2d-oct
> _______________________________________________
> curl-loader-devel mailing list
> cur...@li...
> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel
>
>
> ***  Please note that this message and any attachments may contain confidential
> and proprietary material and information and are intended only for the use of
> the intended recipient(s). If you are not the intended recipient, you are hereby
> notified that any review, use, disclosure, dissemination, distribution or copying
> of this message and any attachments is strictly prohibited. If you have received
> this email in error, please immediately notify the sender and destroy this e-mail
> and any attachments and all copies, whether electronic or printed.
> Please also note that any views, opinions, conclusions or commitments expressed
> in this message are those of the individual sender and do not necessarily reflect
> the views of Fortinet, Inc., its affiliates, and emails are not binding on
> Fortinet and only a writing manually signed by Fortinet's General Counsel can be
> a binding commitment of Fortinet to Fortinet's customers or partners. Thank you. ***
>
> ------------------------------------------------------------------------------
> All the data continuously generated in your IT infrastructure contains a
> definitive record of customers, application performance, security
> threats, fraudulent activity and more. Splunk takes this data and makes
> sense of it. Business sense. IT sense. Common sense.
> http://p.sf.net/sfu/splunk-d2d-oct
> _______________________________________________
> curl-loader-devel mailing list
> cur...@li...
> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel
>
>



-- 
Regards,
Robert Iakobashvili, Ph.D.

Home:       http://www.ghotit.com
Blog:         http://dyslexia-blog.ghotit.com
Twitter:      http://twitter.com/ghotit
Facebook: http://facebook.com/ghotit
......................................................................
                   Ghotit Dyslexia
Assistive technology that understands you
......................................................................