Re: HTTPS throughput too low

[wei g. <wei...@gm...>]

hi,all:
do you have any conclusion for this issue？ I ran into the same problem,
I'll look for answers for this.

On Wed, Oct 12, 2011 at 4:51 PM, Robert Iakobashvili <cor...@gm...>wrote:

> Hi CK Zhang,
>
> Now it clear what happens.
>
> curl-loader:
> - establishes TCP connections;
> - makes expensive TLS/SSL handshake
> - fetches the file.
>
> After fetching file, curl-loader closes the TCP-connection.
>
> Next cycle it establishes the TCP-connection and makes the expensive
> TLS/SSL handshake again,
> and fetches the file.
>
> To achieve high throughput libcurl needs to be configured to re-use
> the established
> TCP connection and TLS/SSL for the next file transfer and not to
> re-establish it every cycle.
>
> I suggest you to look in curl (libcurl) options and I will also be doing
> that.
>
> Thanks for your reporting.
>
> On Wed, Oct 12, 2011 at 2:59 AM, CK Zhang <ck...@fo...> wrote:
> >
> > Attached logs, and output from console
> >
> > ============  loading batch is: Apach
> > ======================================
> >
> ----------------------------------------------------------------------------
> > ----
> > Interval stats (latest:3 sec, clients:100, CAPS-curr:60):
> > H/F
> >
> Req:0,1xx:0,2xx:0,3xx:0,4xx:0,5xx:0,Err:0,T-Err:0,D:0ms,D-2xx:0ms,Ti:0B/s,To
> > :0B/s
> > H/F/S
> >
> Req:200,1xx:0,2xx:201,3xx:0,4xx:0,5xx:0,Err:0,T-Err:0,D:553ms,D-2xx:553ms,Ti
> > :6854220B/s,To:18800B/s
> >
> ----------------------------------------------------------------------------
> > ----
> > Summary stats (runs:48 secs, CAPS-average:60):
> > H/F
> >
> Req:0,1xx:0,2xx:0,3xx:0,4xx:0,5xx:0,Err:0,T-Err:0,D:0ms,D-2xx:0ms,Ti:0B/s,To
> > :0B/s
> > H/F/S
> >
> Req:2878,1xx:0,2xx:2873,3xx:0,4xx:0,5xx:0,Err:0,T-Err:0,D:614ms,D-2xx:614ms,
> > Ti:6087722B/s,To:17182B/s
> >
> ============================================================================
> > =====
> >  Manual: clients:max[100],curr[100]. Inc num: [+|*].
> >
> ============================================================================
> > =====
> >
> > Best Regards,
> > CK
> > #8042
> >
> >
> > -----Original Message-----
> > From: Robert Iakobashvili [mailto:cor...@gm...]
> > Sent: Tuesday, October 11, 2011 5:49 PM
> > To: curl-loader-devel
> > Subject: Re: HTTPS throughput too low
> >
> > Please, try
> >
> > CLIENTS_NUM_MAX=100
> > CLIENTS_NUM_START=100
> >
> > and comment out
> > #CLIENTS_RAMPUP_INC=10
> >
> > Say, what happens with traffic and send me the log.
> >
> > On Tue, Oct 11, 2011 at 11:36 AM, CK Zhang <ck...@fo...> wrote:
> >> Attached :) thanks!
> >>
> >> Best Regards,
> >> CK
> >> #8042
> >>
> >>
> >> -----Original Message-----
> >> From: Robert Iakobashvili [mailto:cor...@gm...]
> >> Sent: Tuesday, October 11, 2011 5:33 PM
> >> To: curl-loader-devel
> >> Subject: Re: HTTPS throughput too low
> >>
> >> Could you send me it zipped not rarred?
> >> Thanks!
> >>
> >> On Tue, Oct 11, 2011 at 11:30 AM, CK Zhang <ck...@fo...>
> wrote:
> >>>  Dear Robert
> >>>
> >>> Attached test log by command "./curl-loader -f https.conf -v"
> >>>
> >>> Best Regards,
> >>> CK
> >>> #8042
> >>>
> >>> -----Original Message-----
> >>> From: Robert Iakobashvili [mailto:cor...@gm...]
> >>> Sent: Tuesday, October 11, 2011 5:19 PM
> >>> To: curl-loader-devel
> >>> Subject: Re: HTTPS throughput too low
> >>>
> >>> Dear CK Zhang,
> >>>
> >>> Can you run HTTPS sessions with -v detailed logging and say if
> >>> you seen in logs something interesting.
> >>> Thanks,
> >>> Robert
> >>>
> >>>
> >>>
> >>> On Tue, Oct 11, 2011 at 11:14 AM, CK Zhang <ck...@fo...>
> wrote:
> >>>> Dear Robert
> >>>>
> >>>>
> >>>>
> >>>> Thanks a lot for your replay,
> >>>>
> >>>>
> >>>>
> >>>> Here are some answer about your questions
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> What happens if you try HTTP? Does it impact throughput?
> >>>>
> >>>>  It is normal and perfect if I test with HTTP, The throughput can
> reach
> >> to
> >>>> 950Mbps.
> >>>>
> >>>>
> >>>>
> >>>> Which tools you are using to measure throughput?
> >>>>
> >>>> There are 2 ways I used,
> >>>>
> >>>>  1 check the log summary form cur-loader like this”
> >>>> Ti:5261372B/s,To:18375B/s”
> >>>>
> >>>> 2 check real time traffic speed on switch with connected test PC and
> > HTTP
> >>>> server
> >>>>
> >>>>
> >>>>
> >>>> What happens, if you try just to ftp some large file between the
> >> machines?
> >>>>
> >>>> NO test
> >>>>
> >>>> What about CPU usage at both client and server?
> >>>>
> >>>> Both they are busy, about 80%
> >>>>
> >>>> Which HTTP server is in use?
> >>>>
> >>>>  Apache/2.2.21
> >>>>
> >>>> Have you looked into logs of curl-loader and your web-server?
> >>>>
> >>>> Checked log, not find abnormal issue.
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> Today, I used Loadrunner11 to simulate https traffic with same
> >>> environments
> >>>> (same apache server) , it can reach to 110Mpbs. But still only get
> >>>  800kbps
> >>>> with cur-loader.
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> Best Regards,
> >>>>
> >>>> CK
> >>>>
> >>>> #8042
> >>>>
> >>>>
> >>>>
> >>>> From: Robert Iakobashvili [mailto:cor...@gm...]
> >>>> Sent: Tuesday, September 27, 2011 1:19 PM
> >>>> To: curl-loader-devel
> >>>> Subject: Re: HTTPS throughput too low
> >>>>
> >>>>
> >>>>
> >>>> Dear CK Zhang,
> >>>>
> >>>> On Mon, Sep 26, 2011 at 5:01 AM, CK Zhang <ck...@fo...>
> wrote:
> >>>>
> >>>>
> >>>>
> >>>> I want to get high https stream with curl loader, but the Https
> >> throughput
> >>>> is about 800kbps between two PC, Http throughput can reach 950Mbps in
> >> that
> >>> 2
> >>>> pcs, I do know if some have experience like this, and get a high https
> >>>> throughput with curl-loader, Thanks for any supports and help!!
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> Here is my configuration:
> >>>>
> >>>>
> >>>>
> >>>> PC1:CPU Inter E2200, Mem 2G, 1000M NIC
> >>>>
> >>>> OpenSSL:
> >>>>
> >>>> openssl-devel-1.0.0-4.el6_0.2.x86_64
> >>>>
> >>>> openssl-1.0.0-4.el6_0.2.x86_64
> >>>>
> >>>> openssl-perl-1.0.0-4.el6_0.2.x86_64
> >>>>
> >>>>
> >>>>
> >>>> curl-loader-0.53
> >>>>
> >>>>                                ########### GENERAL SECTION
> >>>> ################################
> >>>>
> >>>> BATCH_NAME= Apach
> >>>>
> >>>> CLIENTS_NUM_MAX=400
> >>>>
> >>>> CLIENTS_NUM_START=100
> >>>>
> >>>> CLIENTS_RAMPUP_INC=10
> >>>>
> >>>> INTERFACE   =eth2
> >>>>
> >>>> NETMASK=16
> >>>>
> >>>> IP_ADDR_MIN= 172.16.76.150
> >>>>
> >>>> IP_ADDR_MAX= 172.16.246.250  #Actually - this is for self-control
> >>>>
> >>>> CYCLES_NUM= -1
> >>>>
> >>>> URLS_NUM= 1
> >>>>
> >>>>
> >>>>
> >>>> ########### URL SECTION ####################################
> >>>>
> >>>>
> >>>>
> >>>> URL=https://172.16.76.142/was/100KB.txt # the size of 100KB.txt is
> 100KB
> >>>>
> >>>> #URL=http://localhost/ACE-INSTALL.html
> >>>>
> >>>> URL_SHORT_NAME="local-index"
> >>>>
> >>>> REQUEST_TYPE=GET
> >>>>
> >>>> TIMER_URL_COMPLETION = 0      # In msec. When positive, Now it is
> >> enforced
> >>>> by cancelling url fetch on timeout
> >>>>
> >>>> TIMER_AFTER_URL_SLEEP =1000
> >>>>
> >>>> ~
> >>>>
> >>>>
> >>>>
> >>>> PC2. Works as Webserver,CPU Intel E7600,Mem 4G, 1000M NIC
> >>>>
> >>>> Server version: Apache/2.2.21 (Unix)
> >>>>
> >>>> Server built:   Sep 21 2011 16:19:43
> >>>>
> >>>> openssl-devel-1.0.0-4.el6_0.2.x86_64
> >>>>
> >>>> openssl-1.0.0-4.el6_0.2.x86_64
> >>>>
> >>>> openssl-perl-1.0.0-4.el6_0.2.x86_64
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> Interesting.
> >>>>
> >>>>
> >>>>
> >>>> What happens if you try HTTP? Does it impact throughput?
> >>>>
> >>>>
> >>>>
> >>>> Which tools you are using to measure throughput?
> >>>>
> >>>>
> >>>>
> >>>> What happens, if you try just to ftp some large file between the
> >> machines?
> >>>>
> >>>>
> >>>>
> >>>> What about CPU usage at both client and server?
> >>>>
> >>>> Which HTTP server is in use?
> >>>>
> >>>>
> >>>>
> >>>> Have you looked into logs of curl-loader and your web-server?
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> Regards,
> >>>> Robert Iakobashvili, Ph.D.
> >>>>
> >>>>
> >>>>
> >>>> Home:       http://www.ghotit.com
> >>>>
> >>>> Blog:         http://dyslexia-blog.ghotit.com
> >>>>
> >>>> Twitter:      http://twitter.com/ghotit
> >>>>
> >>>> Facebook: http://facebook.com/ghotit
> >>>>
> >>>> ......................................................................
> >>>>
> >>>>                    Ghotit Dyslexia
> >>>>
> >>>> Assistive technology that understands you
> >>>>
> >>>> ......................................................................
> >>>>
> >>>>
> >>>>
> >>>> ________________________________
> >>>>
> >>>> ***  Please note that this message and any attachments may contain
> >>>> confidential
> >>>> and proprietary material and information and are intended only for the
> >> use
> >>>> of
> >>>> the intended recipient(s). If you are not the intended recipient, you
> > are
> >>>> hereby
> >>>> notified that any review, use, disclosure, dissemination, distribution
> > or
> >>>> copying
> >>>> of this message and any attachments is strictly prohibited. If you
> have
> >>>> received
> >>>> this email in error, please immediately notify the sender and destroy
> >> this
> >>>> e-mail
> >>>> and any attachments and all copies, whether electronic or printed.
> >>>> Please also note that any views, opinions, conclusions or commitments
> >>>> expressed
> >>>> in this message are those of the individual sender and do not
> > necessarily
> >>>> reflect
> >>>> the views of Fortinet, Inc., its affiliates, and emails are not
> binding
> >> on
> >>>> Fortinet and only a writing manually signed by Fortinet's General
> > Counsel
> >>>> can be
> >>>> a binding commitment of Fortinet to Fortinet's customers or partners.
> >>> Thank
> >>>> you. ***
> >>>>
> >>>> ________________________________
> >>>>
> >>>>
> >>>
> >>
> >
> ----------------------------------------------------------------------------
> >>> --
> >>>> All the data continuously generated in your IT infrastructure
> contains a
> >>>> definitive record of customers, application performance, security
> >>>> threats, fraudulent activity and more. Splunk takes this data and
> makes
> >>>> sense of it. Business sense. IT sense. Common sense.
> >>>> http://p.sf.net/sfu/splunk-d2d-oct
> >>>> _______________________________________________
> >>>> curl-loader-devel mailing list
> >>>> cur...@li...
> >>>> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel
> >>>>
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> Regards,
> >>> Robert Iakobashvili, Ph.D.
> >>>
> >>> Home:       http://www.ghotit.com
> >>> Blog:         http://dyslexia-blog.ghotit.com
> >>> Twitter:      http://twitter.com/ghotit
> >>> Facebook: http://facebook.com/ghotit
> >>> ......................................................................
> >>>                    Ghotit Dyslexia
> >>> Assistive technology that understands you
> >>> ......................................................................
> >>>
> >>>
> >>
> >
> ----------------------------------------------------------------------------
> >>> --
> >>> All the data continuously generated in your IT infrastructure contains
> a
> >>> definitive record of customers, application performance, security
> >>> threats, fraudulent activity and more. Splunk takes this data and makes
> >>> sense of it. Business sense. IT sense. Common sense.
> >>> http://p.sf.net/sfu/splunk-d2d-oct
> >>> _______________________________________________
> >>> curl-loader-devel mailing list
> >>> cur...@li...
> >>> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel
> >>>
> >>>
> >>> ***  Please note that this message and any attachments may contain
> >> confidential
> >>> and proprietary material and information and are intended only for the
> > use
> >> of
> >>> the intended recipient(s). If you are not the intended recipient, you
> are
> >> hereby
> >>> notified that any review, use, disclosure, dissemination, distribution
> or
> >> copying
> >>> of this message and any attachments is strictly prohibited. If you have
> >> received
> >>> this email in error, please immediately notify the sender and destroy
> > this
> >> e-mail
> >>> and any attachments and all copies, whether electronic or printed.
> >>> Please also note that any views, opinions, conclusions or commitments
> >> expressed
> >>> in this message are those of the individual sender and do not
> necessarily
> >> reflect
> >>> the views of Fortinet, Inc., its affiliates, and emails are not binding
> > on
> >>> Fortinet and only a writing manually signed by Fortinet's General
> Counsel
> >> can be
> >>> a binding commitment of Fortinet to Fortinet's customers or partners.
> >> Thank you. ***
> >>>
> >>>
> >>
> >
> ----------------------------------------------------------------------------
> >> --
> >>> All the data continuously generated in your IT infrastructure contains
> a
> >>> definitive record of customers, application performance, security
> >>> threats, fraudulent activity and more. Splunk takes this data and makes
> >>> sense of it. Business sense. IT sense. Common sense.
> >>> http://p.sf.net/sfu/splunk-d2d-oct
> >>> _______________________________________________
> >>> curl-loader-devel mailing list
> >>> cur...@li...
> >>> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel
> >>>
> >>>
> >>
> >>
> >>
> >> --
> >> Regards,
> >> Robert Iakobashvili, Ph.D.
> >>
> >> Home:       http://www.ghotit.com
> >> Blog:         http://dyslexia-blog.ghotit.com
> >> Twitter:      http://twitter.com/ghotit
> >> Facebook: http://facebook.com/ghotit
> >> ......................................................................
> >>                    Ghotit Dyslexia
> >> Assistive technology that understands you
> >> ......................................................................
> >>
> >>
> >
> ----------------------------------------------------------------------------
> >> --
> >> All the data continuously generated in your IT infrastructure contains a
> >> definitive record of customers, application performance, security
> >> threats, fraudulent activity and more. Splunk takes this data and makes
> >> sense of it. Business sense. IT sense. Common sense.
> >> http://p.sf.net/sfu/splunk-d2d-oct
> >> _______________________________________________
> >> curl-loader-devel mailing list
> >> cur...@li...
> >> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel
> >>
> >>
> >> ***  Please note that this message and any attachments may contain
> > confidential
> >> and proprietary material and information and are intended only for the
> use
> > of
> >> the intended recipient(s). If you are not the intended recipient, you
> are
> > hereby
> >> notified that any review, use, disclosure, dissemination, distribution
> or
> > copying
> >> of this message and any attachments is strictly prohibited. If you have
> > received
> >> this email in error, please immediately notify the sender and destroy
> this
> > e-mail
> >> and any attachments and all copies, whether electronic or printed.
> >> Please also note that any views, opinions, conclusions or commitments
> > expressed
> >> in this message are those of the individual sender and do not
> necessarily
> > reflect
> >> the views of Fortinet, Inc., its affiliates, and emails are not binding
> on
> >> Fortinet and only a writing manually signed by Fortinet's General
> Counsel
> > can be
> >> a binding commitment of Fortinet to Fortinet's customers or partners.
> > Thank you. ***
> >>
> >>
> >
> ----------------------------------------------------------------------------
> > --
> >> All the data continuously generated in your IT infrastructure contains a
> >> definitive record of customers, application performance, security
> >> threats, fraudulent activity and more. Splunk takes this data and makes
> >> sense of it. Business sense. IT sense. Common sense.
> >> http://p.sf.net/sfu/splunk-d2d-oct
> >> _______________________________________________
> >> curl-loader-devel mailing list
> >> cur...@li...
> >> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel
> >>
> >>
> >
> >
> >
> > --
> > Regards,
> > Robert Iakobashvili, Ph.D.
> >
> > Home:       http://www.ghotit.com
> > Blog:         http://dyslexia-blog.ghotit.com
> > Twitter:      http://twitter.com/ghotit
> > Facebook: http://facebook.com/ghotit
> > ......................................................................
> >                    Ghotit Dyslexia
> > Assistive technology that understands you
> > ......................................................................
> >
> >
> ----------------------------------------------------------------------------
> > --
> > All the data continuously generated in your IT infrastructure contains a
> > definitive record of customers, application performance, security
> > threats, fraudulent activity and more. Splunk takes this data and makes
> > sense of it. Business sense. IT sense. Common sense.
> > http://p.sf.net/sfu/splunk-d2d-oct
> > _______________________________________________
> > curl-loader-devel mailing list
> > cur...@li...
> > https://lists.sourceforge.net/lists/listinfo/curl-loader-devel
> >
> >
> > ***  Please note that this message and any attachments may contain
> confidential
> > and proprietary material and information and are intended only for the
> use of
> > the intended recipient(s). If you are not the intended recipient, you
> are hereby
> > notified that any review, use, disclosure, dissemination, distribution
> or copying
> > of this message and any attachments is strictly prohibited. If you have
> received
> > this email in error, please immediately notify the sender and destroy
> this e-mail
> > and any attachments and all copies, whether electronic or printed.
> > Please also note that any views, opinions, conclusions or commitments
> expressed
> > in this message are those of the individual sender and do not
> necessarily reflect
> > the views of Fortinet, Inc., its affiliates, and emails are not binding
> on
> > Fortinet and only a writing manually signed by Fortinet's General
> Counsel can be
> > a binding commitment of Fortinet to Fortinet's customers or partners.
> Thank you. ***
> >
> >
> ------------------------------------------------------------------------------
> > All the data continuously generated in your IT infrastructure contains a
> > definitive record of customers, application performance, security
> > threats, fraudulent activity and more. Splunk takes this data and makes
> > sense of it. Business sense. IT sense. Common sense.
> > http://p.sf.net/sfu/splunk-d2d-oct
> > _______________________________________________
> > curl-loader-devel mailing list
> > cur...@li...
> > https://lists.sourceforge.net/lists/listinfo/curl-loader-devel
> >
> >
>
>
>
> --
> Regards,
> Robert Iakobashvili, Ph.D.
>
> Home:       http://www.ghotit.com
> Blog:         http://dyslexia-blog.ghotit.com
> Twitter:      http://twitter.com/ghotit
> Facebook: http://facebook.com/ghotit
> ......................................................................
>                    Ghotit Dyslexia
> Assistive technology that understands you
> ......................................................................
>
>
> ------------------------------------------------------------------------------
> All the data continuously generated in your IT infrastructure contains a
> definitive record of customers, application performance, security
> threats, fraudulent activity and more. Splunk takes this data and makes
> sense of it. Business sense. IT sense. Common sense.
> http://p.sf.net/sfu/splunk-d2d-oct
> _______________________________________________
> curl-loader-devel mailing list
> cur...@li...
> https://lists.sourceforge.net/lists/listinfo/curl-loader-devel
>