OpenVPN

On Fri, Nov 18, 2011 at 8:41 PM, Diogo Melo <dmelo87@...> wrote:

> Hi,
>
>
> I wonder if someone have rough estimates about how many connections a
> generic server (Inter i5 2GB RAM memory) would be able to keep/handle (i
> know that it depends on a lot of other variables but rough is enough).
>
>
Theoretically, an Intel I5 2.3ghz machine with 2gb ram, and 500gb disk
space (Running Linux OS, with 2GB Swap space) is capable of handling +200
connections.

The problem comes in at the initialisation of the connection. if 200
machines had to establish a vpn connection at the same time, your box will
be dead.



> If that hypothetical server is able to handle X connections, then would 10
> servers be able to handle 10X connections or this doesn't grows linearly?
>
>From the bottom of my heart, what I really need to know (or at least have a
> clue) is how much hardware I would need to keep 15000 connections up,
> considering that each connection will transfer just a small amount of data
> (lets say 50KB per HOUR for another computer inside the network)?
>

No, the connections don't grow linearly,.
The connections are limited by  number of factors, including, but not
limited to server hardware.
You need to keep in mind bandwidth requirements, and distance from VPN
client to VPN server, as additional routing can cause lower TTL.
Also

I suggest a 1 : 63 ratio.
That is 63 clients to every server
It even splits your subnets into /24 networks

Using the above ratio, you looking at roughly 240 machines to handle 15000
connections.
And this does not include redundancy.

Is the data mission critical?
Is there a DR site?
Or are you planning for DR VPN Servers? (VPN Cluster?)


>
> Any help would be very appreciated.
>
>
> Diogo Oliveira de Melo
> Ciência da Computação
> ICMC - USP São Carlos
>
>
>
>
> ------------------------------------------------------------------------------
> All the data continuously generated in your IT infrastructure
> contains a definitive record of customers, application performance,
> security threats, fraudulent activity, and more. Splunk takes this
> data and makes sense of it. IT sense. And common sense.
> http://p.sf.net/sfu/splunk-novd2d
> _______________________________________________
> Openvpn-users mailing list
> Openvpn-users@...
> https://lists.sourceforge.net/lists/listinfo/openvpn-users
>
>


-- 
We live in an age when pizza gets to your home before the police.
  - Jeff Marder
------------------------------------------
Allan Swanepoel
allanice001@...
allanice.001@...
dragonmaster@...
+27 71 850 5554
Linux User #452990
Linux Machine #360914
-----------------------------------------------
IMPORTANT: This email is intended for the use of  the  individual
addressee(s)  named  above  and  may  contain information that is
confidential,  privileged  or  unsuitable  for  overly  sensitive persons
with  low  self-esteem, no sense of humour or irrational religious beliefs.
If you are not  the  intended  recipient,  any dissemination,
distribution  or copying  of  this  email is not authorised (either
explicitly or implicitly) and  constitutes  an irritating  social  faux
pas. Unless the word absquatulation has been used in its correct context
somewhere  other  than  in  this warning, it does not have any legal or
grammatical use and may be ignored. No animals were  harmed  in  the
transmission  of  this email,  although  the  yorkshire  terrier  next door
is living on borrowed time, let me tell you. Those of you with an
overwhelming fear  of  the unknown will be gratified to learn that there is
no hidden message revealed by reading  this  warning  backwards,  so just
ignore that Alert Notice from Microsoft: However, by pouring a complete
circle of salt around yourself and your  computer  you can  ensure  that
no harm befalls you and your pets.

Diogo Oliveira de Melo
Ciência da Computação
ICMC - USP São Carlos




On Sat, Nov 19, 2011 at 12:17 AM, Diogo Melo <dmelo87@...> wrote:

> Hi,
>
>
> thanks a lot for the answer, Allan. I'm now considering using two Cisco
> ASA 5585-X with SSP-20 to do the job. Do you have any thoughts about that?
>
> The mission is not so critical. 99.9% update is enough, since the client
> machines can cache it's data and send it later on. There is no planning for
> a DR site right now but redundancy is suitable. I saw that Cisco ASA comes
> with redundant power supply and cooler so i'm not sure it would be helpful
> to get a third machine.
>
> It will be either a VPN cluster or two Cisco ASA working in parallel. I
> think the second option is more suitable not only thinking about overall
> price of the equipment but also on datacenter resources consumption. Each
> Cisco ASA occupy only 2U and surely much less power supply.
>
> I know Cisco ASA doesn't use OpenVPN but do you have any experience using
> it as a VPN server? Any thoughts about disadvantages?
>
>
>
> Diogo Oliveira de Melo
> Ciência da Computação
> ICMC - USP São Carlos
>
>
>
>
> On Fri, Nov 18, 2011 at 9:10 PM, Allan Swanepoel <allanice001@...:
>
>>
>>
>> On Fri, Nov 18, 2011 at 8:41 PM, Diogo Melo <dmelo87@...> wrote:
>>
>>> Hi,
>>>
>>>
>>> I wonder if someone have rough estimates about how many connections a
>>> generic server (Inter i5 2GB RAM memory) would be able to keep/handle (i
>>> know that it depends on a lot of other variables but rough is enough).
>>>
>>>
>> Theoretically, an Intel I5 2.3ghz machine with 2gb ram, and 500gb disk
>> space (Running Linux OS, with 2GB Swap space) is capable of handling +200
>> connections.
>>
>> The problem comes in at the initialisation of the connection. if 200
>> machines had to establish a vpn connection at the same time, your box will
>> be dead.
>>
>>
>>
>>> If that hypothetical server is able to handle X connections, then would
>>> 10 servers be able to handle 10X connections or this doesn't grows linearly?
>>>
>> From the bottom of my heart, what I really need to know (or at least have
>>> a clue) is how much hardware I would need to keep 15000 connections up,
>>> considering that each connection will transfer just a small amount of data
>>> (lets say 50KB per HOUR for another computer inside the network)?
>>>
>>
>> No, the connections don't grow linearly,.
>> The connections are limited by  number of factors, including, but not
>> limited to server hardware.
>> You need to keep in mind bandwidth requirements, and distance from VPN
>> client to VPN server, as additional routing can cause lower TTL.
>> Also
>>
>> I suggest a 1 : 63 ratio.
>> That is 63 clients to every server
>> It even splits your subnets into /24 networks
>>
>> Using the above ratio, you looking at roughly 240 machines to handle
>> 15000 connections.
>> And this does not include redundancy.
>>
>> Is the data mission critical?
>> Is there a DR site?
>> Or are you planning for DR VPN Servers? (VPN Cluster?)
>>
>>
>>>
>>> Any help would be very appreciated.
>>>
>>>
>>> Diogo Oliveira de Melo
>>> Ciência da Computação
>>> ICMC - USP São Carlos
>>>
>>>
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> All the data continuously generated in your IT infrastructure
>>> contains a definitive record of customers, application performance,
>>> security threats, fraudulent activity, and more. Splunk takes this
>>> data and makes sense of it. IT sense. And common sense.
>>> http://p.sf.net/sfu/splunk-novd2d
>>> _______________________________________________
>>> Openvpn-users mailing list
>>> Openvpn-users@...
>>> https://lists.sourceforge.net/lists/listinfo/openvpn-users
>>>
>>>
>>
>>
>> --
>> We live in an age when pizza gets to your home before the police.
>>   - Jeff Marder
>> ------------------------------------------
>> Allan Swanepoel
>> allanice001@...
>> allanice.001@...
>> dragonmaster@...
>> +27 71 850 5554
>> Linux User #452990
>> Linux Machine #360914
>> -----------------------------------------------
>> IMPORTANT: This email is intended for the use of  the  individual
>> addressee(s)  named  above  and  may  contain information that is
>> confidential,  privileged  or  unsuitable  for  overly  sensitive persons
>> with  low  self-esteem, no sense of humour or irrational religious beliefs.
>> If you are not  the  intended  recipient,  any dissemination,
>> distribution  or copying  of  this  email is not authorised (either
>> explicitly or implicitly) and  constitutes  an irritating  social  faux
>> pas. Unless the word absquatulation has been used in its correct context
>> somewhere  other  than  in  this warning, it does not have any legal or
>> grammatical use and may be ignored. No animals were  harmed  in  the
>> transmission  of  this email,  although  the  yorkshire  terrier  next door
>> is living on borrowed time, let me tell you. Those of you with an
>> overwhelming fear  of  the unknown will be gratified to learn that there is
>> no hidden message revealed by reading  this  warning  backwards,  so just
>> ignore that Alert Notice from Microsoft: However, by pouring a complete
>> circle of salt around yourself and your  computer  you can  ensure  that
>> no harm befalls you and your pets.
>>
>
>

On Sat, Nov 19, 2011 at 4:23 AM, Tom Kunz
<tkunz@...:

> Somehow 1:63 sounds painfully low.   Just one single instance of ovpn with
> the /30 topology? Yikes, I've done VM's with a handful of /24's on them.
> I'm thinking probably 1k+ ovpn clients for a modest machine.  Obviously not
> all starting simultaneously, but the load will be exceptionally low,
> basically nonexistent, for a mere 60 ovpn clients.
>

That is very low. if you have decent hardware, i would suggest
virtualization, but then i don't see Intel i5 with 2gb ram as "decent".
I've seen what happens to machines with 2GB ram at +- 100 client
connections. CPU starts running haywire, and disk activity starts kicking
into overdrive, since the system is now starting to utilise a lot more swap
space.

The reason that happens, is open vpn re-checks the connected ssl certs
every 30 minutes or so, pushing the cpu through the roof. even if you don't
initialize all the clients together, you start spiking cpu at 60 machines.

On Sat, 19 Nov 2011 00:17:57 -0200, Diogo Melo <dmelo87@...> wrote:
>
>
>
> Diogo Oliveira de Melo
> Ciência da Computação
> ICMC - USP São Carlos
>
>
>
>
> On Sat, Nov 19, 2011 at 12:17 AM, Diogo Melo <dmelo87@...> wrote:
>
>> Hi,
>>  thanks a lot for the answer, Allan. I'm now considering using two Cisco
>> ASA 5585-X with SSP-20 to do the job. Do you have any thoughts about that?
>>
>
Personally, i HATE Cisco. They are way over priced, and will never over you
a true ssl vpn solution


>  The mission is not so critical. 99.9% update is enough, since the client
>> machines can cache it's data and send it later on. There is no planning for
>> a DR site right now but redundancy is suitable. I saw that Cisco ASA comes
>> with redundant power supply and cooler so i'm not sure it would be helpful
>> to get a third machine.
>>  It will be either a VPN cluster or two Cisco ASA working in parallel. I
>> think the second option is more suitable not only thinking about overall
>> price of the equipment but also on datacenter resources consumption. Each
>> Cisco ASA occupy only 2U and surely much less power supply.
>>  I know Cisco ASA doesn't use OpenVPN but do you have any experience
>> using it as a VPN server? Any thoughts about disadvantages?
>>
>
the biggest problem is the ram. push the ram to 8gb - 16gb, and yes,
virtualisation just became an option.

Throw 2 machines like that together, and the solution has just become a
completely different ball game.
If you can, use vsphere / hypervm as a virtualization stack. very stable.
My personal choice would be to use openvz, since the virtualization stack
is plugged into the kernel, and a lot less resource intensive than any
vmware product.

Now with 16gb ram, run 3 -5 servers, at 3 - 4gb ram each, and you can still
run openvpn on the host os, although that would in my opinion become too
risky, as it may cause the vm's to crash once usage kicks in.

with this, you could run +- 350 connections per virtual instance. your cpu
will still take a beating,but disk IO won't be that heavy.


>
>>
>> Diogo Oliveira de Melo
>> Ciência da Computação
>> ICMC - USP São Carlos
>>
>>
>>
>>
>>  On Fri, Nov 18, 2011 at 9:10 PM, Allan Swanepoel <allanice001@...:
>>
>>>
>>>
>>>  On Fri, Nov 18, 2011 at 8:41 PM, Diogo Melo <dmelo87@...> wrote:
>>>
>>>> Hi,
>>>>  I wonder if someone have rough estimates about how many connections a
>>>> generic server (Inter i5 2GB RAM memory) would be able to keep/handle (i
>>>> know that it depends on a lot of other variables but rough is enough).
>>>>
>>>
>>> Theoretically, an Intel I5 2.3ghz machine with 2gb ram, and 500gb disk
>>> space (Running Linux OS, with 2GB Swap space) is capable of handling +200
>>> connections.
>>>
>>> The problem comes in at the initialisation of the connection. if 200
>>> machines had to establish a vpn connection at the same time, your box will
>>> be dead.
>>>
>>>
>>>
>>>>  If that hypothetical server is able to handle X connections, then
>>>> would 10 servers be able to handle 10X connections or this doesn't grows
>>>> linearly?
>>>>
>>>  >From the bottom of my heart, what I really need to know (or at least
>>>> have a clue) is how much hardware I would need to keep 15000 connections
>>>> up, considering that each connection will transfer just a small amount of
>>>> data (lets say 50KB per HOUR for another computer inside the network)?
>>>>
>>>
>>> No, the connections don't grow linearly,.
>>> The connections are limited by  number of factors, including, but not
>>> limited to server hardware.
>>> You need to keep in mind bandwidth requirements, and distance from VPN
>>> client to VPN server, as additional routing can cause lower TTL.
>>> Also
>>>
>>> I suggest a 1 : 63 ratio.
>>> That is 63 clients to every server
>>> It even splits your subnets into /24 networks
>>>
>>> Using the above ratio, you looking at roughly 240 machines to handle
>>> 15000 connections.
>>> And this does not include redundancy.
>>>
>>> Is the data mission critical?
>>> Is there a DR site?
>>> Or are you planning for DR VPN Servers? (VPN Cluster?)
>>>
>>>
>>>>   Any help would be very appreciated.
>>>>
>>>>
>>>> Diogo Oliveira de Melo
>>>> Ciência da Computação
>>>> ICMC - USP São Carlos
>>>>
>>>>
>>>>
>>>>
>>>> ------------------------------------------------------------------------------
>>>> All the data continuously generated in your IT infrastructure
>>>> contains a definitive record of customers, application performance,
>>>> security threats, fraudulent activity, and more. Splunk takes this
>>>> data and makes sense of it. IT sense. And common sense.
>>>> http://p.sf.net/sfu/splunk-novd2d
>>>> _______________________________________________
>>>> Openvpn-users mailing list
>>>> Openvpn-users@...
>>>> https://lists.sourceforge.net/lists/listinfo/openvpn-users
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> We live in an age when pizza gets to your home before the police.
>>>   - Jeff Marder
>>> ------------------------------------------
>>> Allan Swanepoel
>>> allanice001@...
>>> allanice.001@...
>>> dragonmaster@...
>>> +27 71 850 5554
>>> Linux User #452990
>>> Linux Machine #360914
>>> -----------------------------------------------
>>> IMPORTANT: This email is intended for the use of  the  individual
>>> addressee(s)  named  above  and  may  contain information that is
>>> confidential,  privileged  or  unsuitable  for  overly  sensitive persons
>>> with  low  self-esteem, no sense of humour or irrational religious beliefs.
>>> If you are not  the  intended  recipient,  any dissemination,
>>> distribution  or copying  of  this  email is not authorised (either
>>> explicitly or implicitly) and  constitutes  an irritating  social  faux
>>> pas. Unless the word absquatulation has been used in its correct context
>>> somewhere  other  than  in  this warning, it does not have any legal or
>>> grammatical use and may be ignored. No animals were  harmed  in  the
>>> transmission  of  this email,  although  the  yorkshire  terrier  next door
>>> is living on borrowed time, let me tell you. Those of you with an
>>> overwhelming fear  of  the unknown will be gratified to learn that there is
>>> no hidden message revealed by reading  this  warning  backwards,  so just
>>> ignore that Alert Notice from Microsoft: However, by pouring a complete
>>> circle of salt around yourself and your  computer  you can  ensure  that
>>> no harm befalls you and your pets.
>>>
>>
>
>



-- 
We live in an age when pizza gets to your home before the police.
  - Jeff Marder
------------------------------------------
Allan Swanepoel
allanice001@...
allanice.001@...
dragonmaster@...
+27 71 850 5554
Linux User #452990
Linux Machine #360914
-----------------------------------------------
IMPORTANT: This email is intended for the use of  the  individual
addressee(s)  named  above  and  may  contain information that is
confidential,  privileged  or  unsuitable  for  overly  sensitive persons
with  low  self-esteem, no sense of humour or irrational religious beliefs.
If you are not  the  intended  recipient,  any dissemination,
distribution  or copying  of  this  email is not authorised (either
explicitly or implicitly) and  constitutes  an irritating  social  faux
pas. Unless the word absquatulation has been used in its correct context
somewhere  other  than  in  this warning, it does not have any legal or
grammatical use and may be ignored. No animals were  harmed  in  the
transmission  of  this email,  although  the  yorkshire  terrier  next door
is living on borrowed time, let me tell you. Those of you with an
overwhelming fear  of  the unknown will be gratified to learn that there is
no hidden message revealed by reading  this  warning  backwards,  so just
ignore that Alert Notice from Microsoft: However, by pouring a complete
circle of salt around yourself and your  computer  you can  ensure  that
no harm befalls you and your pets.

On 19/11/11 17:41, Allan Swanepoel wrote:
>
> The reason that happens, is open vpn re-checks the connected ssl certs
> every 30 minutes or so, pushing the cpu through the roof. even if you
> don't initialize all the clients together, you start spiking cpu at 60
> machines.

That is manageable via the "--reneg*" options. If you have lots of
hosts, you could move it out to 10 hours without much reduction in
security - I think most of Cisco's VPN stuff renegotiates keys every 8
hours(?).

If you have i5 or i7 CPUs, can the hardware AES chipset help improve
performance?

-- 
Cheers

Jason Haar
Information Security Manager, Trimble Navigation Ltd.
Phone: +64 3 9635 377 Fax: +64 3 9635 417
PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1

On Sat, Nov 19, 2011 at 3:58 AM, Jason Haar <Jason.Haar@...> wrote:
> On 19/11/11 17:41, Allan Swanepoel wrote:
>>
>> The reason that happens, is open vpn re-checks the connected ssl certs
>> every 30 minutes or so, pushing the cpu through the roof. even if you
>> don't initialize all the clients together, you start spiking cpu at 60
>> machines.
>
> That is manageable via the "--reneg*" options. If you have lots of
> hosts, you could move it out to 10 hours without much reduction in
> security - I think most of Cisco's VPN stuff renegotiates keys every 8
> hours(?).
>
> If you have i5 or i7 CPUs, can the hardware AES chipset help improve
> performance?
>

What happens if you have a circuit glitch of some sort that makes the
clients disconnect?  Is there a way to make their retries back off
gracefully?

-- 
  Les Mikesell
    lesmikesell@...

* Allan Swanepoel <allanice001@...>:

> I suggest a 1 : 63 ratio.
> That is 63 clients to every server
> It even splits your subnets into /24 networks

We have 100 simultaneous clients on our VPN server, and we have a 5
minute load average of 0.03. My clients are mostly browsing the web
(free medical full textes and journals via the library)

At the same time the whole system uses about 500MB of RAM. Meaning
that our limiting factor is the RAM. That's about 5MB per user.

-- 
Ralf Hildebrandt
  Geschäftsbereich IT | Abteilung Netzwerk
  Charité - Universitätsmedizin Berlin
  Campus Benjamin Franklin
  Hindenburgdamm 30 | D-12203 Berlin
  Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962
  ralf.hildebrandt@... | http://www.charite.de

> I've seen what happens to machines with 2GB ram at +- 100 client
> connections. CPU starts running haywire, and disk activity starts kicking
> into overdrive, since the system is now starting to utilise a lot more swap
> space.

We have 100 clients on a 2GB machine and basically nothing happens. 5
minute load average 0.03

> The reason that happens, is open vpn re-checks the connected ssl certs
> every 30 minutes or so, pushing the cpu through the roof. even if you don't
> initialize all the clients together, you start spiking cpu at 60 machines.

That can be configured! 
reneg-sec 36000
(we set the value to 10h, that way the lenght of a typical session
never execeeds the renegotiation interval)


-- 
Ralf Hildebrandt
  Geschäftsbereich IT | Abteilung Netzwerk
  Charité - Universitätsmedizin Berlin
  Campus Benjamin Franklin
  Hindenburgdamm 30 | D-12203 Berlin
  Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962
  ralf.hildebrandt@... | http://www.charite.de

> -----Original Message-----
> From: Jason Haar [mailto:Jason.Haar@...]
> Sent: Saturday, November 19, 2011 1:58 AM
> To: openvpn-users@...
> Subject: Re: [Openvpn-users] Scalability
> 
> On 19/11/11 17:41, Allan Swanepoel wrote:
> >
> > The reason that happens, is open vpn re-checks the connected ssl
> certs
> > every 30 minutes or so, pushing the cpu through the roof. even if you
> > don't initialize all the clients together, you start spiking cpu at
> 60
> > machines.
> 
> That is manageable via the "--reneg*" options. If you have lots of
> hosts, you could move it out to 10 hours without much reduction in
> security - I think most of Cisco's VPN stuff renegotiates keys every 8
> hours(?).
> 
> If you have i5 or i7 CPUs, can the hardware AES chipset help improve
> performance?
> 

Yes the Intel AES in the newer chips vastly improves performance IF you use openssl 1.0.0 or newer. I currently maintain some VPN servers that have the following specs:

Dual Intel 5160's (Dual Core 3Ghz)
4GB RAM

I'm running 3 OpenVPN Daemons per server as OpenVPN is single threaded. Each VPN instance can easily handle 1000 active connections. And has on many occasions. Each daemon can handle about 340Mb. On the newer Intel 56xx CPU's I tested with that have the AES hardware using 2.93Ghz 6-core CPU's and OpenSSL 1.0.0d I was able to achieve closer to 460Mb. This was using UDP. So I believe a single larger server could handle upwards of 10,000 connections. But this will all depend on what kind of average throughput. 

Here's the encrypt I use and some basic config:

topology subnet
push "topology subnet"
proto udp
keepalive 10 60
comp-lzo
user nobody
group nobody
duplicate-cn
persist-key
persist-tun
verb 3
cipher AES-256-CBC
auth SHA512
tls-cipher DHE-RSA-AES256-SHA
port 1194
dev tun0

I use port 1195 as well as additional IP's per server. This is running on CentOS 5.2. 

Hope this helps.

Brad


This email may contain confidential and privileged material for the sole use of the intended recipient. Any review, use, distribution or disclosure by others is strictly prohibited. If you are not the intended recipient (or authorized to receive for the recipient), please contact the sender by reply email and delete all copies of this message.

On 20/11/11 05:45, Les Mikesell wrote:
> What happens if you have a circuit glitch of some sort that makes the
> clients disconnect? Is there a way to make their retries back off
> gracefully? 
I was wondering about that too. Some form of "--retry-random-wait"
option may be the best way of dealing with that? Obviously it would mean
clients would hang for longer periods of time before getting a tunnel
back up - a "lose-lose" situation :-(

Are there ways of reducing the load caused by key renegotiation? Why
does that hit the CPU so much anyway? 

-- 
Cheers

Jason Haar
Information Security Manager, Trimble Navigation Ltd.
Phone: +64 3 9635 377 Fax: +64 3 9635 417
PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1

On Sat, Nov 19, 2011 at 7:15 PM, Ralf Hildebrandt <
Ralf.Hildebrandt@...> wrote:

>
> > I've seen what happens to machines with 2GB ram at +- 100 client
> > connections. CPU starts running haywire, and disk activity starts kicking
> > into overdrive, since the system is now starting to utilise a lot more
> swap
> > space.
>
> We have 100 clients on a 2GB machine and basically nothing happens. 5
> minute load average 0.03
>
>
Well, i spoke of my experience with a single board computer, housing a i5
2.3ghz CPU, with 2gb ram and a 160gb sata disk.

I used the same machine with 4gb ram, and saw a notable difference



> > The reason that happens, is open vpn re-checks the connected ssl certs
> > every 30 minutes or so, pushing the cpu through the roof. even if you
> don't
> > initialize all the clients together, you start spiking cpu at 60
> machines.
>
> That can be configured!
> reneg-sec 36000
> (we set the value to 10h, that way the lenght of a typical session
> never execeeds the renegotiation interval)
>
>
> --
> Ralf Hildebrandt
>  Geschäftsbereich IT | Abteilung Netzwerk
>  Charité - Universitätsmedizin Berlin
>  Campus Benjamin Franklin
>  Hindenburgdamm 30 | D-12203 Berlin
>  Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962
>  ralf.hildebrandt@... | http://www.charite.de
>
>
>
> ------------------------------------------------------------------------------
> All the data continuously generated in your IT infrastructure
> contains a definitive record of customers, application performance,
> security threats, fraudulent activity, and more. Splunk takes this
> data and makes sense of it. IT sense. And common sense.
> http://p.sf.net/sfu/splunk-novd2d
> _______________________________________________
> Openvpn-users mailing list
> Openvpn-users@...
> https://lists.sourceforge.net/lists/listinfo/openvpn-users
>



-- 
We live in an age when pizza gets to your home before the police.
  - Jeff Marder
------------------------------------------
Allan Swanepoel
allanice001@...
allanice.001@...
dragonmaster@...
+27 71 850 5554
Linux User #452990
Linux Machine #360914
-----------------------------------------------
IMPORTANT: This email is intended for the use of  the  individual
addressee(s)  named  above  and  may  contain information that is
confidential,  privileged  or  unsuitable  for  overly  sensitive persons
with  low  self-esteem, no sense of humour or irrational religious beliefs.
If you are not  the  intended  recipient,  any dissemination,
distribution  or copying  of  this  email is not authorised (either
explicitly or implicitly) and  constitutes  an irritating  social  faux
pas. Unless the word absquatulation has been used in its correct context
somewhere  other  than  in  this warning, it does not have any legal or
grammatical use and may be ignored. No animals were  harmed  in  the
transmission  of  this email,  although  the  yorkshire  terrier  next door
is living on borrowed time, let me tell you. Those of you with an
overwhelming fear  of  the unknown will be gratified to learn that there is
no hidden message revealed by reading  this  warning  backwards,  so just
ignore that Alert Notice from Microsoft: However, by pouring a complete
circle of salt around yourself and your  computer  you can  ensure  that
no harm befalls you and your pets.

* Allan Swanepoel <allanice001@...>:

> > We have 100 clients on a 2GB machine and basically nothing happens. 5
> > minute load average 0.03
> >
> >
> Well, i spoke of my experience with a single board computer, housing a i5
> 2.3ghz CPU, with 2gb ram and a 160gb sata disk.
> 
> I used the same machine with 4gb ram, and saw a notable difference

Maybe it depends on what the users do. MY users MAINLY use the VPN to
surf the net (full text PDFs from PubMed). Some users use remote desktops
(vmware VDI)

-- 
Ralf Hildebrandt
  Geschäftsbereich IT | Abteilung Netzwerk
  Charité - Universitätsmedizin Berlin
  Campus Benjamin Franklin
  Hindenburgdamm 30 | D-12203 Berlin
  Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962
  ralf.hildebrandt@... | http://www.charite.de

* Jason Haar <Jason.Haar@...>:

> I was wondering about that too. Some form of "--retry-random-wait"
> option may be the best way of dealing with that?

No, it's not needed, since both the server and client are checking if
the opposite side is still available. Thus (at least with UDP) the
reconnection times ARE already randomized within the timeperiods given
by the following parameters:

keepalive / ping / ping-restart

> Obviously it would mean clients would hang for longer periods of time
> before getting a tunnel back up - a "lose-lose" situation :-(

Yep.
 
> Are there ways of reducing the load caused by key renegotiation? Why
> does that hit the CPU so much anyway? 

The key renegotiation is a CPU intensive operation, since public key
algorithms are involved. Once the sides agreed on a key, that key is
being used for a cpu-unintensive symmetrical encryption.

-- 
Ralf Hildebrandt
  Geschäftsbereich IT | Abteilung Netzwerk
  Charité - Universitätsmedizin Berlin
  Campus Benjamin Franklin
  Hindenburgdamm 30 | D-12203 Berlin
  Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962
  ralf.hildebrandt@... | http://www.charite.de

On Sun, Nov 20, 2011 at 12:04:59PM +0100, Ralf Hildebrandt wrote:

> > does that hit the CPU so much anyway? 
> 
> The key renegotiation is a CPU intensive operation, since public key
> algorithms are involved. Once the sides agreed on a key, that key is
> being used for a cpu-unintensive symmetrical encryption.

Is AMD Interlagos (Bulldozer) doing better or worse than Sandy Bridge
for SSL/OpenVPN at the moment, anyone knows?

-- 
Eugen* Leitl <a href="http://leitl.org">leitl</a> http://leitl.org
______________________________________________________________
ICBM: 48.07100, 11.36820 http://www.ativel.com http://postbiota.org
8B29F6BE: 099D 78BA 2FD3 B014 B08A  7779 75B0 2443 8B29 F6BE

* Eugen Leitl <eugen@...>:
> On Sun, Nov 20, 2011 at 12:04:59PM +0100, Ralf Hildebrandt wrote:
> 
> > > does that hit the CPU so much anyway? 
> > 
> > The key renegotiation is a CPU intensive operation, since public key
> > algorithms are involved. Once the sides agreed on a key, that key is
> > being used for a cpu-unintensive symmetrical encryption.
> 
> Is AMD Interlagos (Bulldozer) doing better or worse than Sandy Bridge
> for SSL/OpenVPN at the moment, anyone knows?

I'd like to know this as well :)
Maybe one could make a benchmark, but there are more problems:

* can older tunnelblick/openvpn clients handle new(er) ciphers?
* are config changes on the client side required?

-- 
Ralf Hildebrandt
  Geschäftsbereich IT | Abteilung Netzwerk
  Charité - Universitätsmedizin Berlin
  Campus Benjamin Franklin
  Hindenburgdamm 30 | D-12203 Berlin
  Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962
  ralf.hildebrandt@... | http://www.charite.de

Hi Ralf, *,

Ralf Hildebrandt wrote:
> * Eugen Leitl <eugen@...>:
>   
>> On Sun, Nov 20, 2011 at 12:04:59PM +0100, Ralf Hildebrandt wrote:
>>
>>     
>>>> does that hit the CPU so much anyway? 
>>>>         
>>> The key renegotiation is a CPU intensive operation, since public key
>>> algorithms are involved. Once the sides agreed on a key, that key is
>>> being used for a cpu-unintensive symmetrical encryption.
>>>       
>> Is AMD Interlagos (Bulldozer) doing better or worse than Sandy Bridge
>> for SSL/OpenVPN at the moment, anyone knows?
>>     
>
> I'd like to know this as well :)
> Maybe one could make a benchmark, but there are more problems:
>   
as soon as I can get my hands on an AMD Bulldozer I'll test this; I've
been measuring openssl/aesni as well as openvpn performance for quite
some time now. I've measured the output of 'openssl speed -evp
aes-256-cbc' and some other commands and put them all in a large
spreadsheet, normalized for clockspeed. Some results:
- BlowFish (BF) is almost fully clockspeed dependent; only i7's and Xeon
Xnnnn's show a 15 percent speedup
- AES-256 without aes-ni support is also fully clockspeed dependent; see
above
- an AESNI capable (Intel) CPU is roughly 8 times faster when
encrypting/decrypting small packets (< 512 bytes) ;
- for larger packets (>= 1024 bytes) the speed gain drops to a factor of 4.
Some other interesting tidbits
- 'turboboost' speeds up things a bit as long as your CPU is not doing
much else : TurboBoost allows one core to run at the higher clockspeed,
but as soon as the other cores/threads need to chip in, the clockspeed
drops to the "nominal" rate again.
- the RHEL/Centos 5 openssl-fips slows down openssl by a factor of 2 !
- recompiling openssl with the flag 'noasm' (that is, don't use the
handcoded assembly files provided by openssl) speeds up AES-256 on the
non-aesni capable CPUs by a factor of 2 for small packets!

Clearly the encryption routines that openssl provides can be tuned for
more efficiency.
All tests were done on Linux (CentOS , Fedora) ; when I ran the same
test on a dual boot machine (Windows 7 64bit vs Fedora 14 64bit) I
noticed a 5% speed gain for the non-aesni tests on *windows* : the
windows compiled version of openssl that comes with OpenVPN 2.2 is
slightly faster than the gcc-compiled version on Linux. However, the
Windows version of openssl does *NOT* make use of aes-ni capable CPUs,
I've added this as a feature request for the next release of OpenVPN.

I hope to get a SandyBridge2 machine soon (yes, they're not released
yet), to see how well it fares, and I hope to get a bulldozer for
testing as well.

Note that these numbers are "raw" openssl tests only; openvpn
performance is quite a different thing. For an OpenVPN server it makes
sense to run multiple OpenVPN instances on different ports, in order to
spread the load.
The Windows OpenVPN client/tap-win32 driver is *still* much less
efficient than the Linux/*BSD version; two students of mine did some
tests on FreeBSD and saw that it was quite a bit faster than Linux when
handling high bandwidth clients.

> * can older tunnelblick/openvpn clients handle new(er) ciphers?
> * are config changes on the client side required?
>
>   
openvpn 2.0+ supports the AES-256 cipher on all platforms, IIRC, but the
version of OpenSSL used in that version most likely will not support the
aes-ni cpu instructions. For that a recompilation of the underlying
OpenSSL libs is required.

Other than specifying
  cipher AES-256-CBC
no changes should be required.

cheers,

JJK

* Jan Just Keijser <janjust@...>:

> openvpn 2.0+ supports the AES-256 cipher on all platforms, IIRC, but the
> version of OpenSSL used in that version most likely will not support the
> aes-ni cpu instructions. For that a recompilation of the underlying
> OpenSSL libs is required.
> 
> Other than specifying
>   cipher AES-256-CBC
> no changes should be required.

I tried changing the cipher and a TunnelBlick client failed:

Nov 21 09:59:07 vpn-ausfall ovpn-server[7937]: 109.45.0.21:30731 WARNING: 'cipher' is used inconsistently, local='cipher AES-256-CBC', remote='cipher BF-CBC'
Nov 21 09:59:07 vpn-ausfall ovpn-server[7937]: 109.45.0.21:30731 WARNING: 'keysize' is used inconsistently, local='keysize 256', remote='keysize 128'

Note that the TunnelBlick config does NOT (!) have any cipher or
keysize directives.

-- 
Ralf Hildebrandt
  Geschäftsbereich IT | Abteilung Netzwerk
  Charité - Universitätsmedizin Berlin
  Campus Benjamin Franklin
  Hindenburgdamm 30 | D-12203 Berlin
  Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962
  ralf.hildebrandt@... | http://www.charite.de

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 21/11/11 10:01, Ralf Hildebrandt wrote:
> * Jan Just Keijser <janjust@...>:
> 
>> openvpn 2.0+ supports the AES-256 cipher on all platforms, IIRC, but
>> the version of OpenSSL used in that version most likely will not
>> support the aes-ni cpu instructions. For that a recompilation of the
>> underlying OpenSSL libs is required.
>> 
>> Other than specifying cipher AES-256-CBC no changes should be
>> required.
> 
> I tried changing the cipher and a TunnelBlick client failed:
> 
> Nov 21 09:59:07 vpn-ausfall ovpn-server[7937]: 109.45.0.21:30731
> WARNING: 'cipher' is used inconsistently, local='cipher AES-256-CBC',
> remote='cipher BF-CBC' Nov 21 09:59:07 vpn-ausfall ovpn-server[7937]:
> 109.45.0.21:30731 WARNING: 'keysize' is used inconsistently,
> local='keysize 256', remote='keysize 128'
> 
> Note that the TunnelBlick config does NOT (!) have any cipher or 
> keysize directives.

Well, both client and server need to use the same --cipher and --keysize.
 Some --ciphers cannot use --keysize, as the key size is fixed in the
chosen cipher.  If you do 'openvpn --show-ciphers' you will see all
supported ciphers, and if the key size is fixed or variable.

AFAIK, TunnelBlick uses the "stock" openvpn binary - it's just a OSX GUI
for OpenVPN.  So if it's possible to edit the config files manually, it
is possible to use any supported cipher/key size.

But as always, the supported ciphers/key sizes is also depending on what
the local OpenSSL library supports as well.


kind regards,

David Sommerseth
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk7KGtkACgkQDC186MBRfrqHiwCePLFkEu2a20gqh3B0aiU2NRGN
VRoAniUCldYJS9iGr+9+TGK7cNbKfYlq
=mv6Y
-----END PGP SIGNATURE-----

* David Sommerseth <openvpn.list@...>:

> Well, both client and server need to use the same --cipher and --keysize.
>  Some --ciphers cannot use --keysize, as the key size is fixed in the
> chosen cipher.  If you do 'openvpn --show-ciphers' you will see all
> supported ciphers, and if the key size is fixed or variable.
> 
> AFAIK, TunnelBlick uses the "stock" openvpn binary - it's just a OSX GUI
> for OpenVPN.  So if it's possible to edit the config files manually, it
> is possible to use any supported cipher/key size.

Yes, but my point is that I'd have to change ALL config files out
there in the wild :|

-- 
Ralf Hildebrandt
  Geschäftsbereich IT | Abteilung Netzwerk
  Charité - Universitätsmedizin Berlin
  Campus Benjamin Franklin
  Hindenburgdamm 30 | D-12203 Berlin
  Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962
  ralf.hildebrandt@... | http://www.charite.de

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 21/11/11 11:21, Ralf Hildebrandt wrote:
> * David Sommerseth <openvpn.list@...>:
> 
>> Well, both client and server need to use the same --cipher and
>> --keysize. Some --ciphers cannot use --keysize, as the key size is
>> fixed in the chosen cipher.  If you do 'openvpn --show-ciphers' you
>> will see all supported ciphers, and if the key size is fixed or
>> variable.
>> 
>> AFAIK, TunnelBlick uses the "stock" openvpn binary - it's just a OSX
>> GUI for OpenVPN.  So if it's possible to edit the config files
>> manually, it is possible to use any supported cipher/key size.
> 
> Yes, but my point is that I'd have to change ALL config files out 
> there in the wild :|

Yes, that is correct.  I see that as an issue too, the server should be
able to negotiate protocols - according to the best one the client
supports.  However, that's something we need to pull into the roadmap for
OpenVPN 3.


kind regards,

David Sommerseth

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk7KJ/EACgkQDC186MBRfrptxACgnsryE4t+BJgMUFk80ZJLdlKI
dMMAn3ksC/YUQ+KyQyeVkuuoKAk0GN2Y
=LI7d
-----END PGP SIGNATURE-----

Ralf Hildebrandt wrote:
> * David Sommerseth <openvpn.list@...>:
>
>   
>> Well, both client and server need to use the same --cipher and --keysize.
>>  Some --ciphers cannot use --keysize, as the key size is fixed in the
>> chosen cipher.  If you do 'openvpn --show-ciphers' you will see all
>> supported ciphers, and if the key size is fixed or variable.
>>
>> AFAIK, TunnelBlick uses the "stock" openvpn binary - it's just a OSX GUI
>> for OpenVPN.  So if it's possible to edit the config files manually, it
>> is possible to use any supported cipher/key size.
>>     
>
> Yes, but my point is that I'd have to change ALL config files out
> there in the wild :|
>
>   
this is not a feature but it might work: what happens if you add
  push "cipher AES-256-CBC"
to the server config?

JJK

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 21/11/11 11:35, Jan Just Keijser wrote:
> Ralf Hildebrandt wrote:
>> * David Sommerseth <openvpn.list@...>:
>> 
> this is not a feature but it might work: what happens if you add push
> "cipher AES-256-CBC" to the server config?

I doubt that will work, as the PUSH options are pushed _after_ the tunnel
has been initialised.  And in this scenario, OpenVPN would have to do a
complete disconnect+reconnect again to get the new tunnel parameters up.
 I'm even doubtful 'cipher' is pushable.

Unless somebody proves me wrong ... I've not tried it nor checked the
source code.


kind regards,

David Sommerseth
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk7KK5IACgkQDC186MBRfrqlbwCeM+/Ste5+LLtxu6m542ZLmNzi
zToAniPh5UBLkEk4Ez8bdXXj3jRBz+KW
=UiUk
-----END PGP SIGNATURE-----

Hi David,

David Sommerseth wrote:
> On 21/11/11 11:35, Jan Just Keijser wrote:
> > Ralf Hildebrandt wrote:
> >> * David Sommerseth <openvpn.list@...>:
> >>
> > this is not a feature but it might work: what happens if you add push
> > "cipher AES-256-CBC" to the server config?
>
> I doubt that will work, as the PUSH options are pushed _after_ the tunnel
> has been initialised.  And in this scenario, OpenVPN would have to do a
> complete disconnect+reconnect again to get the new tunnel parameters up.
>  I'm even doubtful 'cipher' is pushable.
>
> Unless somebody proves me wrong ... I've not tried it nor checked the
> source code.
>

I hate to admit it, but you're right ;)
Just tested it here, and you can't push "cipher" ...

cheers,

JJK

> > I was wondering about that too. Some form of "--retry-random-wait"
> > option may be the best way of dealing with that?
> 
> No, it's not needed, since both the server and client are checking if
> the opposite side is still available. Thus (at least with UDP) the
> reconnection times ARE already randomized within the timeperiods
> given
> by the following parameters:
> 
> keepalive / ping / ping-restart

I wouldn't see that as an effective substitute.  Any number of problems toward the server side will still compel all the clients to reconnect shortly after the server becomes available again.  This re-synchronizes every client's reneg-sec cycle to near lock-step with each other. The server will again be hammered with renegotiation work every reneg-sec seconds and have no renegotiation work to do for the long intervals between.  

I'm working with a lab that has only 2000 concurrent connections, and this happens every time the server is rebooted or faced with any other disruption.  The reneg cycles are then synchronized and stay synchronized unless individual client disruptions manage to stagger those clients.  I'm experimenting with rate-limiting the new OpenVPN connections per second with iptables rules, which feels really kludgy, to help spread the load.  

I'd expect the scalability to improve with the "--retry-random-wait" parameter to prevent the clients from all jumping back on at the same time.  

Failing that, or in addition, something like a "--reneg-diffusion n" on the server could add some randomness to each client's reneg-sec cycle and spread the renegotiation load in instances when the clients' initial connections are synchronized.

So what part of the renegotiation is so CPU-intensive? Is it the key
generation, or the TLS session over which the new key is transmitted the
"heavy lifting" part?

If it's the TLS layer that's expensive, what about extending it's
lifespan and reusing it? e.g. have a 24hour TLS lifespan, but
renegotiate data channel keys more often. I *think* the TLS session is
aggressively encrypted, and used only to exchange the keys over (and do
the cert validation of course). I'd imagine very little data is sent
over it, so increasing its lifespan shouldn't really increase
brute-forcing opportunities much (or at least give people the option?)

Is there a nice description/diagram somewhere showing all the "chatter"
that goes on during an openvpn session?  I for one am prattling on based
on very little knowledge ;-)

-- 
Cheers

Jason Haar
Information Security Manager, Trimble Navigation Ltd.
Phone: +64 3 9635 377 Fax: +64 3 9635 417
PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 22/11/11 01:50, Jason Haar wrote:
> So what part of the renegotiation is so CPU-intensive? Is it the key 
> generation, or the TLS session over which the new key is transmitted
> the "heavy lifting" part?

SSL/TLS is kind of tricky.  To make public/private key pairs safe, you
need pretty big public keys.  Nowadays, 2048 or 4096 bits are not
unusual.  Working with such big keys and the algorithm expenses related
to PKI encryption takes quite some CPU time.  It's the mathematics behind
it which is hefty.

Symmetrical encryption (same password on both sides), are on the other
hand very cheap and fast.  So what SSL/TLS does, is to negotiate through
different Key Exchange algorithms (KEX) a shared secret.  So when the
SSL/TLS handshake is completed, both sides have a randomized shared
"password" which is used for the symmetrical encryption.

But do you really want to have a static password "forever"?  If someone
manages to crack that secret, they can decrypt everything.  Which is why
there is a key renegotiation.  In OpenVPN, this is controlled by the
- --reneg-* options.  This forces a password change of shared secret.
However, such a change initialises yet another CPU intensive KEX.

If you google around, you'll find several comparisons between key sizes
of asymmetrical (PKI) and symmetrical encryption algorithms.  IIRC, 4096
bits PKI is pretty close to 256bits symmetric encryption - in encryption
_strength_.

So both are roughly equally strong, but symmetric is way faster than
asymmetric encryption.  However, with PKI you have greater flexibility,
as everyone can have your public key (which is in the server certificates
sent to clients) - but none of them can decrypt your secrets (easily).
While in the symmetric world, everyone got the same encryption and
decryption key.  Which makes asymmetric encryption initially more secure
if you don't ultimately trust all your clients.

> If it's the TLS layer that's expensive, what about extending it's 
> lifespan and reusing it? e.g. have a 24hour TLS lifespan, but 
> renegotiate data channel keys more often. I *think* the TLS session
> is aggressively encrypted, and used only to exchange the keys over
> (and do the cert validation of course). I'd imagine very little data
> is sent over it, so increasing its lifespan shouldn't really increase 
> brute-forcing opportunities much (or at least give people the
> option?)

I don't recall the details exactly here, but IIRC, everything is using
the same temporary symmetric key.  The only moment where the PKI keys
comes into play are when the KEX process starts, and a new temporary
symmetric encryption key is negotiated.  So by extending the lifespan of
the symmetric keys, you gradually weaken the security of the VPN tunnel.

> Is there a nice description/diagram somewhere showing all the
> "chatter" that goes on during an openvpn session?  I for one am
> prattling on based on very little knowledge ;-)

I learnt a lot when I read "SSL and TLS, Designing and Building Secure
Systems" by Eric Rescorla [1].  It's getting dated now, but the initial
concepts are the same - and it describes the different SSL/TLS levels,
how the handshake process works and the whole KEX process for RSA and
DSS, including the Diffie-Hellman algorithm.  Eric Rescorla is also one
of those persons behind the SSL/TLS protocol.  I know some people
disagree with me on how good this book is, but I found it good.

Another book which I have /not/ read, but which looks pretty good at
first glance is "Implementing SSL/TLS Using Cryptography and PKI" by
Joshua Davies [2].

That's SSL/TLS stuff.  For OpenVPN, it's basically the same.  The only
difference OpenVPN has done is to make SSL/TLS work over UDP.  SSL/TLS is
designed for TCP.  But as we know, VPN over TCP is much less efficient
than over UDP (ref: TCP in TCP issues).  In 2006 an RFC describing DTLS,
which solves SSL/TLS over UDP was published, but at that point OpenVPN
was roughly 4 years old already.  OpenVPN have because of this not
changed the protocol itself.

How OpenVPN does the SSL/TLS over UDP, is basically to intercept the
outgoing SSL stream, wrap it into a container which contains some
reliability data (packet sequence numbers, etc) and send it through a UDP
socket.  On the receiving side, it's a similar logic, which also takes
care of the ordering of packets as well.  When OpenVPN is used in TCP
mode, it will still use the same wrapper functionality as well.  Which is
also why OpenVPN traffic isn't detected as "proper" SSL/TLS traffic.  But
except of that extra wrapping, it is basically just standard SSL/TLS.


kind regards,

David Sommerseth


[1]
<http://www.amazon.com/SSL-TLS-Designing-Building-Systems/dp/0201615983/ref=sr_1_3?ie=UTF8&qid=1321966357&sr=8-3>
[2]
<http://www.amazon.com/Implementing-SSL-TLS-Using-Cryptography/dp/0470920416/ref=sr_1_1?ie=UTF8&qid=1321966357&sr=8-1>
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk7LnpMACgkQDC186MBRfrohPQCfSzptRz3c2XDM3raOjawQKd/Q
LK0An2f/03l/Gj5AHcoK4iveyEfy2eUm
=9A5c
-----END PGP SIGNATURE-----

Thanks for your description David - it basically matches what I thought.

So here's my next related question: why does key renegotiation hurt
openvpn so much? I run lots of Apache HTTPS servers and never see any
such issues - and yet they use TLS too (and openssl) and so use the same
fundamental technology? If you look at HTTP/1.0 (HTTPS transaction per
URL) vs HTTP/1.1 (reuse HTTPS TCP session for multiple URLs), I'd go as
far as to say the TLS overhead for Apache is much worse than it is for
openvpn - so why isn't this issue seen with Apache (at such small
numbers as I've seen thrown around in this thread). We run HTTPS servers
with larger throughput than our openvpn servers  - they're big servers -
but no SSL hardware accelerators too....

(disclaimer: I don't use openvpn enough to see any of these issues - but
others in this thread have so I'm assuming it's true)

-- 
Cheers

Jason Haar
Information Security Manager, Trimble Navigation Ltd.
Phone: +64 3 9635 377 Fax: +64 3 9635 417
PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1

On Tue, Nov 22, 2011 at 7:07 AM, David Sommerseth
<openvpn.list@...> wrote:
>
> SSL/TLS is kind of tricky.  To make public/private key pairs safe, you
> need pretty big public keys.  Nowadays, 2048 or 4096 bits are not
> unusual.  Working with such big keys and the algorithm expenses related
> to PKI encryption takes quite some CPU time.  It's the mathematics behind
> it which is hefty.

Is there any substantial difference between server/certificate mode
and PTP/shared secret in terms of scaling?  Assuming you don't mind
managing a bunch of separate configs and trust the linux scheduler to
deal with running the processes...

-- 
  Les Mikesell
    lesmikesell@...

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 22/11/11 21:52, Jason Haar wrote:
> So here's my next related question: why does key renegotiation hurt 
> openvpn so much? I run lots of Apache HTTPS servers and never see any 
> such issues - and yet they use TLS too (and openssl) and so use the
> same fundamental technology? If you look at HTTP/1.0 (HTTPS
> transaction per URL) vs HTTP/1.1 (reuse HTTPS TCP session for multiple
> URLs), I'd go as far as to say the TLS overhead for Apache is much
> worse than it is for openvpn - so why isn't this issue seen with
> Apache (at such small numbers as I've seen thrown around in this
> thread). We run HTTPS servers with larger throughput than our openvpn
> servers  - they're big servers - but no SSL hardware accelerators
> too....

Good question!  There is a slightly difference between OpenVPN and Apache
(or most other web servers) with SSL.  The SSL protocol can reuse
negotiated temporary keys.  This is crucially important with HTTPS, as
each element on a web page is downloaded via separate TCP connections.
So, negotiating a new key for each connection to the server would
basically be very heavy for the web server.  This is the same, no matter
if you use HTTP/1.0 or HTTP/1.1 - as the SSL/TLS connection needs to have
been negotiated /before/ the HTTP protocol comes into action.

OpenVPN have this feature disabled, and use forced re-negotiation defined
by --reneg-* options instead.  This means that OpenVPN will never reuse
any temporary keys already negotiated if the SSL/TLS connection needs to
be re-established.  And in fact, that turned out to be pretty good for
OpenVPN, as the CVE-2009-3555 issue in the SSL/TLS protocol didn't have
the same impact on OpenVPN as on web servers.

For a few OpenVPN connections, you will most likely not notice much CPU
hog during the SSL/TLS KEX process.  But if you have some 100 connections
doing this at the very same time, I expect this be noticed.

And this is also why web servers might not be so sensitive to this, as
unless you run a high profile SSL site, you probably don't have too many
simultaneously new SSL/TLS connections being established at the same
time.  In the moment you have that, the load will rise, and in these
cases a SSL hardware accelerators will help.  But if such hardware is
supported by OpenSSL, OpenVPN can also make use of such features.  And
today, with Intel i5 and i7 CPUs with AES-NI support, this works too
(--engine aesni).

Another difference between Apache and OpenVPN, is that Apache is
multi-threaded or multi-processed.  So Apache will spread the load more
evenly out on all CPU cores.  OpenVPN is single-threaded, so it will
impact all other clients connected to this OpenVPN instance.


kind regards,

David Sommerseth
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk7Mp4gACgkQDC186MBRfrrEOQCfQTkuCL7Pd4u/GNAtHtT6ghOK
VdoAnRt9GTYYiMBjK3EatxE+4rng0iW5
=utZ5
-----END PGP SIGNATURE-----

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 23/11/11 00:04, Les Mikesell wrote:
> On Tue, Nov 22, 2011 at 7:07 AM, David Sommerseth 
> <openvpn.list@...> wrote:
>> 
>> SSL/TLS is kind of tricky.  To make public/private key pairs safe,
>> you need pretty big public keys.  Nowadays, 2048 or 4096 bits are
>> not unusual.  Working with such big keys and the algorithm expenses
>> related to PKI encryption takes quite some CPU time.  It's the
>> mathematics behind it which is hefty.
> 
> Is there any substantial difference between server/certificate mode 
> and PTP/shared secret in terms of scaling?  Assuming you don't mind 
> managing a bunch of separate configs and trust the linux scheduler to 
> deal with running the processes...

If you use --secret and not any --tls-{client,server} options, this will
completely move you over to a static shared secret.  In such a
configuration no KEX process will happen, and there will be no change in
the encryption key at all, even between OpenVPN sessions.

If you use --reneg-sec 0, you will completely stop the renegotiation of
keys.  But if OpenVPN reconnects, a new temporary key will be negotiated.

In this perspective, using --tls-{client,server} with --reneg-sec 0,
gives you a higher security on the the VPN channel than using --secret.
Just because in TLS mode temporary symmetric keys comes into play, even
though only negotiated once per connection.  However, if all clients
reconnects after a server fall-out, you will have the CPU hog during the
KEX process.  That will go away with --secret.


kind regards,

David Sommerseth
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk7MqN4ACgkQDC186MBRfrpArgCfRirRq5UgAUj5me64GitGSuQS
PokAmQFSEcOF/gs+0zHI37AzhibZBR5s
=Ub66
-----END PGP SIGNATURE-----

> Hi,
>
>
> I wonder if someone have rough estimates about how many connections a
> generic server (Inter i5 2GB RAM memory) would be able to keep/handle
> (i know that it depends on a lot of other variables but rough is enough).
>
> If that hypothetical server is able to handle X connections, then
> would 10 servers be able to handle 10X connections or this doesn't
> grows linearly?
>
> From the bottom of my heart, what I really need to know (or at least
> have a clue) is how much hardware I would need to keep 15000
> connections up, considering that each connection will transfer just a
> small amount of data (lets say 50KB per HOUR for another computer
> inside the network)?
>
> Any help would be very appreciated.
>

Hi all,

I've been running fairly large-scale performance tests with OpenVPN
2.2.1 servers on Amazon EC2. OpenVPN configuration has been very close
to the default. I've tested 20, 50 and 100 EC2 instances (VMs) with one
OpenVPN client connecting to the OpenVPN server at exactly the same
time. Timing has been synchronized using cron and at. It's not
possible/necessary to synchronize the clocks on the clients as they get
that information from their virtual hosts. All client connection happen
within 1 second, so the timing is fairly accurate.

OpenVPN servers have had 1 OpenVPN server and 1-4 iperf server instances
running. All client iperf instances connected to the iperf servers
through the VPN tunnel, trying to push as much data in as possible.
Iperf server also seems to push data back to the client (~1/3 of
received data). As iperf is running on the VPN server, it consumes it's
share of the CPU and slows things down somewhat, especially on
single-core server instances (e.g. m1.small).

Anyways, the tests have produced some useful information already:

m1.small can handle 20 clients
- connection establishment took ~2 seconds, with CPU at ~100%
- all clients managed to connect to the VPN

m1.small starts to choke at 50 clients
- connection establishment phase took ~10 seconds, with CPU at ~100%
- all clients managed to connect to the VPN
- 7 clients transfered only very small amounts of data (<0,5MB) during
the 60 second iperf test
- longest delay in establishing a connection to the iperf server was 122
seconds

m1.small still works (kind of) with 100 clients
- connection establishment ~11 seconds, with CPU at ~100%
- most clients managed to connect to the VPN (4 seem to have failed)
- 11 clients transfered only very small amounts of data (<0,5MB) during
the 60 second iperf test
- 80 got really poor performance, between 0,01-0,09MB/sec (as expected)
- longest delay in establishing a connection to the iperf server was 213
seconds

m1.large instance instance can handle 50 clients
- connection establishment took 2-3 seconds, with peak CPU load at 11-12%
- longest delay before being able to establish a connection to the iperf
server was 66 seconds
- most iperf clients were able to connect to iperf server fairly quickly

It seems that iperf server's capability of establishing new client
connections drops quickly with CPU load. For best results it should be
on a separate server with very little load. Also, the iperf client
starts it's timer when it's launched, not when it connects to the iperf
server.

---

So, an OpenVPN 2.2.1 server on a m1.small instance can handle 50+
simultaneous client connections fairly well, provided not much is
happening on the server at the same time. Noticeable delay (~8-9
seconds) is involved, however. The m1.large instances can probably
handle a few hundred simultaneous clients connection at the same time.
My latest test seem fairly reliable, as most of the kinks have been
ironed out. I'm thinking of collecting more performance statistics so
that later we can use them to spot regressions or to compare effect of
different OpenVPN configurations (e.g. OpenSSL vs. PolarSSL).

If you have _any_ ideas for OpenVPN performance testing with large
amount of clients, let me know.

For more details look here:

<https://community.openvpn.net/openvpn/wiki/PerformanceTesting>
<https://community.openvpn.net/openvpn/wiki/OpenVPN_Speed> (from JJK)
<http://aws.amazon.com/ec2/instance-types>
<http://iperf.sourceforge.net>

-- 

Samuli Seppänen
Community Manager
OpenVPN Technologies, Inc

irc freenode net: mattock

> If you use --secret and not any --tls-{client,server} 
> options, this will completely move you over to a static 
> shared secret.  In such a configuration no KEX process will 
> happen, and there will be no change in the encryption key at 
> all, even between OpenVPN sessions.
> 
> If you use --reneg-sec 0, you will completely stop the 
> renegotiation of keys.  But if OpenVPN reconnects, a new 
> temporary key will be negotiated.
> 
> In this perspective, using --tls-{client,server} with 
> --reneg-sec 0, gives you a higher security on the the VPN 
> channel than using --secret.
> Just because in TLS mode temporary symmetric keys comes into 
> play, even though only negotiated once per connection.  
> However, if all clients reconnects after a server fall-out, 
> you will have the CPU hog during the KEX process.  That will 
> go away with --secret.
> 
Is there a parameter to say, if the client detects the server is down and it will wait for a random time(the time can be set by the server depends on the client number, like 1 secs for 10 clients and 10 secs for 100 clients) then reconnect?

Regards,

Andy

Folks,

Just imposing myself here, I can't say that I've done quite so much 
empirical testing as you all have but I can throw my two cents, hoping 
it helps you out.

As you're aware of surely benchmarking can be very tricky, specifically 
as you pointed out ensuring that your loading instrumentation is not 
interfering with the tests. I for certain would endeavor to move your 
iPerf processes off of the VPN servers at least of the "server/head end" 
side of things.

Also I will admit that I've never really bothered to test the limits of 
connection count, I've always been
concerned with max throughput. Care would have to be taken I think when 
your combining evaluation of maximum concurrency with maximum CPU load 
testing.

Based upon all this I do have some data points that maybe can help?

I have  a number of installations running 20 or so road warrior 
connections with no issues. These are on Via C7, 1 Ghz embedded systems 
with 256 MB RAM. Now I'll admit these are on broadband connections they 
are not even capable of saturating the system. I've noted ~ that each 
connection takes ~250-300kb of RAM.

These little boxes have AES-128 acceleration hardware on them. In a 
simple straight up Point to Point test with standard configurations they 
can get about 35-40 Mb/s of 128-AES-CBC throughput at max CPU.

I've also done some testing with larger hardware.

This instrumentation has a:

VPN Server: 4 core Intel Xeon E3-1200, and 1 GB Ethernet as the VPN 
"head end", 2 GB RAM

* My head end iperf server is a physicall separate device which I placed 
logically "behind" the VPN server to there were no Iperf processes 
running on the server itself. This iperf machine was an older Dual 
Processor Single Core CPU Dell 1850 ( Xeon 3.66 Ghz)

* My client end servers two of them (are these more of these 1850 Machines).

I ran multiple OVPN processes on all boxes to take advantage of the 
multi-cpu/cores of these systems.

My result essentially are as follows:

Per Core on the E3 system I maxed CPU on a single core at about 400-450 
Mb/s. Adding more loading machines a second core (i.e. a second OVPN 
process) maxed out at just below this level, so it was near linear. At 
this point I didn't have the ability to generate more load and I also 
began to wonder about the network limits as I was running these boxes in 
a single arm configuration so that traffic from iPerf was going in the 
same physical GB Ethernet interface as it was going out the Tun adapter 
using the same physical interface.

However this combined with some other observations had me conclude with 
a fair degree of certainty that at these not immodest hardware levels 
per process limits were ~ 450 Mb/s, and with appropriate configurations 
(i.e. multiple processes serving out different client sub-nets with 
aggregate routing into the VPN server that one couldn't essentially 
create a system that could handle near 1 GB/s (concurrency aside I'll 
admit).

Frankly I cannot imagine that an Amazon small unit would fundamentally 
have issues with 50 or 100 connections but that honestly is a gut feeling.

Not sure this helps, but I like these topics so..

Colin Ryan






On 11-11-23 5:23 AM, Samuli Seppänen wrote:
>> Hi,
>>
>>
>> I wonder if someone have rough estimates about how many connections a
>> generic server (Inter i5 2GB RAM memory) would be able to keep/handle
>> (i know that it depends on a lot of other variables but rough is enough).
>>
>> If that hypothetical server is able to handle X connections, then
>> would 10 servers be able to handle 10X connections or this doesn't
>> grows linearly?
>>
>>  From the bottom of my heart, what I really need to know (or at least
>> have a clue) is how much hardware I would need to keep 15000
>> connections up, considering that each connection will transfer just a
>> small amount of data (lets say 50KB per HOUR for another computer
>> inside the network)?
>>
>> Any help would be very appreciated.
>>
> Hi all,
>
> I've been running fairly large-scale performance tests with OpenVPN
> 2.2.1 servers on Amazon EC2. OpenVPN configuration has been very close
> to the default. I've tested 20, 50 and 100 EC2 instances (VMs) with one
> OpenVPN client connecting to the OpenVPN server at exactly the same
> time. Timing has been synchronized using cron and at. It's not
> possible/necessary to synchronize the clocks on the clients as they get
> that information from their virtual hosts. All client connection happen
> within 1 second, so the timing is fairly accurate.
>
> OpenVPN servers have had 1 OpenVPN server and 1-4 iperf server instances
> running. All client iperf instances connected to the iperf servers
> through the VPN tunnel, trying to push as much data in as possible.
> Iperf server also seems to push data back to the client (~1/3 of
> received data). As iperf is running on the VPN server, it consumes it's
> share of the CPU and slows things down somewhat, especially on
> single-core server instances (e.g. m1.small).
>
> Anyways, the tests have produced some useful information already:
>
> m1.small can handle 20 clients
> - connection establishment took ~2 seconds, with CPU at ~100%
> - all clients managed to connect to the VPN
>
> m1.small starts to choke at 50 clients
> - connection establishment phase took ~10 seconds, with CPU at ~100%
> - all clients managed to connect to the VPN
> - 7 clients transfered only very small amounts of data (<0,5MB) during
> the 60 second iperf test
> - longest delay in establishing a connection to the iperf server was 122
> seconds
>
> m1.small still works (kind of) with 100 clients
> - connection establishment ~11 seconds, with CPU at ~100%
> - most clients managed to connect to the VPN (4 seem to have failed)
> - 11 clients transfered only very small amounts of data (<0,5MB) during
> the 60 second iperf test
> - 80 got really poor performance, between 0,01-0,09MB/sec (as expected)
> - longest delay in establishing a connection to the iperf server was 213
> seconds
>
> m1.large instance instance can handle 50 clients
> - connection establishment took 2-3 seconds, with peak CPU load at 11-12%
> - longest delay before being able to establish a connection to the iperf
> server was 66 seconds
> - most iperf clients were able to connect to iperf server fairly quickly
>
> It seems that iperf server's capability of establishing new client
> connections drops quickly with CPU load. For best results it should be
> on a separate server with very little load. Also, the iperf client
> starts it's timer when it's launched, not when it connects to the iperf
> server.
>
> ---
>
> So, an OpenVPN 2.2.1 server on a m1.small instance can handle 50+
> simultaneous client connections fairly well, provided not much is
> happening on the server at the same time. Noticeable delay (~8-9
> seconds) is involved, however. The m1.large instances can probably
> handle a few hundred simultaneous clients connection at the same time.
> My latest test seem fairly reliable, as most of the kinks have been
> ironed out. I'm thinking of collecting more performance statistics so
> that later we can use them to spot regressions or to compare effect of
> different OpenVPN configurations (e.g. OpenSSL vs. PolarSSL).
>
> If you have _any_ ideas for OpenVPN performance testing with large
> amount of clients, let me know.
>
> For more details look here:
>
> <https://community.openvpn.net/openvpn/wiki/PerformanceTesting>
> <https://community.openvpn.net/openvpn/wiki/OpenVPN_Speed>  (from JJK)
> <http://aws.amazon.com/ec2/instance-types>
> <http://iperf.sourceforge.net>
>
>
>
> This body part will be downloaded on demand.
>
>
> This body part will be downloaded on demand.

Forgot to mention.

My E3 Server has the latest Intel hardware acceleration on it. I put a 
custom OpenSSL build on it with the intel IPP support on it.

Assuming I did that all correctly I didn't really see any real 
performance improvement on raw throughput....but based upon 
converstations I've seen before and the discussion of the initial Key 
negotiation - recently in this thread by David Sommerseth - this kinda 
makes sense that at these CPU levels the bulk encryption really is a 
non-issue, and that hardware accelerations such as Intel IPP might 
really provide the most bang for the buck in exactly these high 
concurrency situations where you have lot's of key negotiation CPU hits.


Colin


----------



Folks,

Just imposing myself here, I can't say that I've done quite so much 
empirical testing as you all have but I can throw my two cents, hoping 
it helps you out.

As you're aware of surely benchmarking can be very tricky, specifically 
as you pointed out ensuring that your loading instrumentation is not 
interfering with the tests. I for certain would endeavor to move your 
iPerf processes off of the VPN servers at least of the "server/head end" 
side of things.

Also I will admit that I've never really bothered to test the limits of 
connection count, I've always been
concerned with max throughput. Care would have to be taken I think when 
your combining evaluation of maximum concurrency with maximum CPU load 
testing.

Based upon all this I do have some data points that maybe can help?

I have  a number of installations running 20 or so road warrior 
connections with no issues. These are on Via C7, 1 Ghz embedded systems 
with 256 MB RAM. Now I'll admit these are on broadband connections they 
are not even capable of saturating the system. I've noted ~ that each 
connection takes ~250-300kb of RAM.

These little boxes have AES-128 acceleration hardware on them. In a 
simple straight up Point to Point test with standard configurations they 
can get about 35-40 Mb/s of 128-AES-CBC throughput at max CPU.

I've also done some testing with larger hardware.

This instrumentation has a:

VPN Server: 4 core Intel Xeon E3-1200, and 1 GB Ethernet as the VPN 
"head end", 2 GB RAM

* My head end iperf server is a physicall separate device which I placed 
logically "behind" the VPN server to there were no Iperf processes 
running on the server itself. This iperf machine was an older Dual 
Processor Single Core CPU Dell 1850 ( Xeon 3.66 Ghz)

* My client end servers two of them (are these more of these 1850 Machines).

I ran multiple OVPN processes on all boxes to take advantage of the 
multi-cpu/cores of these systems.

My result essentially are as follows:

Per Core on the E3 system I maxed CPU on a single core at about 400-450 
Mb/s. Adding more loading machines a second core (i.e. a second OVPN 
process) maxed out at just below this level, so it was near linear. At 
this point I didn't have the ability to generate more load and I also 
began to wonder about the network limits as I was running these boxes in 
a single arm configuration so that traffic from iPerf was going in the 
same physical GB Ethernet interface as it was going out the Tun adapter 
using the same physical interface.

However this combined with some other observations had me conclude with 
a fair degree of certainty that at these not immodest hardware levels 
per process limits were ~ 450 Mb/s, and with appropriate configurations 
(i.e. multiple processes serving out different client sub-nets with 
aggregate routing into the VPN server that one couldn't essentially 
create a system that could handle near 1 GB/s (concurrency aside I'll 
admit).

Frankly I cannot imagine that an Amazon small unit would fundamentally 
have issues with 50 or 100 connections but that honestly is a gut feeling.

Not sure this helps, but I like these topics so..

Colin Ryan






On 11-11-23 5:23 AM, Samuli Seppänen wrote:
>> Hi,
>>
>>
>> I wonder if someone have rough estimates about how many connections a
>> generic server (Inter i5 2GB RAM memory) would be able to keep/handle
>> (i know that it depends on a lot of other variables but rough is enough).
>>
>> If that hypothetical server is able to handle X connections, then
>> would 10 servers be able to handle 10X connections or this doesn't
>> grows linearly?
>>
>>  From the bottom of my heart, what I really need to know (or at least
>> have a clue) is how much hardware I would need to keep 15000
>> connections up, considering that each connection will transfer just a
>> small amount of data (lets say 50KB per HOUR for another computer
>> inside the network)?
>>
>> Any help would be very appreciated.
>>
> Hi all,
>
> I've been running fairly large-scale performance tests with OpenVPN
> 2.2.1 servers on Amazon EC2. OpenVPN configuration has been very close
> to the default. I've tested 20, 50 and 100 EC2 instances (VMs) with one
> OpenVPN client connecting to the OpenVPN server at exactly the same
> time. Timing has been synchronized using cron and at. It's not
> possible/necessary to synchronize the clocks on the clients as they get
> that information from their virtual hosts. All client connection happen
> within 1 second, so the timing is fairly accurate.
>
> OpenVPN servers have had 1 OpenVPN server and 1-4 iperf server instances
> running. All client iperf instances connected to the iperf servers
> through the VPN tunnel, trying to push as much data in as possible.
> Iperf server also seems to push data back to the client (~1/3 of
> received data). As iperf is running on the VPN server, it consumes it's
> share of the CPU and slows things down somewhat, especially on
> single-core server instances (e.g. m1.small).
>
> Anyways, the tests have produced some useful information already:
>
> m1.small can handle 20 clients
> - connection establishment took ~2 seconds, with CPU at ~100%
> - all clients managed to connect to the VPN
>
> m1.small starts to choke at 50 clients
> - connection establishment phase took ~10 seconds, with CPU at ~100%
> - all clients managed to connect to the VPN
> - 7 clients transfered only very small amounts of data (<0,5MB) during
> the 60 second iperf test
> - longest delay in establishing a connection to the iperf server was 122
> seconds
>
> m1.small still works (kind of) with 100 clients
> - connection establishment ~11 seconds, with CPU at ~100%
> - most clients managed to connect to the VPN (4 seem to have failed)
> - 11 clients transfered only very small amounts of data (<0,5MB) during
> the 60 second iperf test
> - 80 got really poor performance, between 0,01-0,09MB/sec (as expected)
> - longest delay in establishing a connection to the iperf server was 213
> seconds
>
> m1.large instance instance can handle 50 clients
> - connection establishment took 2-3 seconds, with peak CPU load at 11-12%
> - longest delay before being able to establish a connection to the iperf
> server was 66 seconds
> - most iperf clients were able to connect to iperf server fairly quickly
>
> It seems that iperf server's capability of establishing new client
> connections drops quickly with CPU load. For best results it should be
> on a separate server with very little load. Also, the iperf client
> starts it's timer when it's launched, not when it connects to the iperf
> server.
>
> ---
>
> So, an OpenVPN 2.2.1 server on a m1.small instance can handle 50+
> simultaneous client connections fairly well, provided not much is
> happening on the server at the same time. Noticeable delay (~8-9
> seconds) is involved, however. The m1.large instances can probably
> handle a few hundred simultaneous clients connection at the same time.
> My latest test seem fairly reliable, as most of the kinks have been
> ironed out. I'm thinking of collecting more performance statistics so
> that later we can use them to spot regressions or to compare effect of
> different OpenVPN configurations (e.g. OpenSSL vs. PolarSSL).
>
> If you have _any_ ideas for OpenVPN performance testing with large
> amount of clients, let me know.
>
> For more details look here:
>
> <https://community.openvpn.net/openvpn/wiki/PerformanceTesting>
> <https://community.openvpn.net/openvpn/wiki/OpenVPN_Speed>  (from JJK)
> <http://aws.amazon.com/ec2/instance-types>
> <http://iperf.sourceforge.net>
>
>
>
> This body part will be downloaded on demand.
>
>
> This body part will be downloaded on demand.

On 24/11/11 06:39, Colin Ryan wrote:
> I ran multiple OVPN processes on all boxes to take advantage of the 
> multi-cpu/cores of these systems.
Care to explain how you do this? Is that using iptables trickery? I bet
there are a lot of people who'd like to know how to do this right :-)

-- 
Cheers

Jason Haar
Information Security Manager, Trimble Navigation Ltd.
Phone: +64 3 9635 377 Fax: +64 3 9635 417
PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1

> -----Original Message-----
> From: Jason Haar [mailto:Jason.Haar@...]
> Sent: Wednesday, November 23, 2011 4:02 PM
> To: openvpn-users@...
> Subject: Re: [Openvpn-users] Scalability
> 
> On 24/11/11 06:39, Colin Ryan wrote:
> > I ran multiple OVPN processes on all boxes to take advantage of the
> > multi-cpu/cores of these systems.
> Care to explain how you do this? Is that using iptables trickery? I bet
> there are a lot of people who'd like to know how to do this right :-)
> 

Multiple ports and IP's on a server. Then in your client config you can use:

remote-random
openvpn.yourdomain.com 1194
openvpn.yourdomain.com 1195

The DNS entry then contains multiple A records. 

This email may contain confidential and privileged material for the sole use of the intended recipient. Any review, use, distribution or disclosure by others is strictly prohibited. If you are not the intended recipient (or authorized to receive for the recipient), please contact the sender by reply email and delete all copies of this message.

I wish, nothing so elegant, and nothing others have not done before.

Each core has:

* An ovpn config file hence process all pointed to the same set of 
certificates.

* Each ovpn config pushes from a subnet of a larger network definition 
or have scripted IP distribution

* An aggregate route is sent to the vpn server

* Remote random connections on the client.

C

On 11-11-23 7:02 PM, Jason Haar wrote:
> On 24/11/11 06:39, Colin Ryan wrote:
>> I ran multiple OVPN processes on all boxes to take advantage of the
>> multi-cpu/cores of these systems.
> Care to explain how you do this? Is that using iptables trickery? I bet
> there are a lot of people who'd like to know how to do this right :-)
>

David Sommerseth wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 22/11/11 21:52, Jason Haar wrote:
>   
>> So here's my next related question: why does key renegotiation hurt 
>> openvpn so much? I run lots of Apache HTTPS servers and never see any 
>> such issues - and yet they use TLS too (and openssl) and so use the
>> same fundamental technology? If you look at HTTP/1.0 (HTTPS
>> transaction per URL) vs HTTP/1.1 (reuse HTTPS TCP session for multiple
>> URLs), I'd go as far as to say the TLS overhead for Apache is much
>> worse than it is for openvpn - so why isn't this issue seen with
>> Apache (at such small numbers as I've seen thrown around in this
>> thread). We run HTTPS servers with larger throughput than our openvpn
>> servers  - they're big servers - but no SSL hardware accelerators
>> too....
>>     
>
> Good question!  There is a slightly difference between OpenVPN and Apache
> (or most other web servers) with SSL.  The SSL protocol can reuse
> negotiated temporary keys.  This is crucially important with HTTPS, as
> each element on a web page is downloaded via separate TCP connections.
> So, negotiating a new key for each connection to the server would
> basically be very heavy for the web server.  This is the same, no matter
> if you use HTTP/1.0 or HTTP/1.1 - as the SSL/TLS connection needs to have
> been negotiated /before/ the HTTP protocol comes into action.
>
> OpenVPN have this feature disabled, and use forced re-negotiation defined
> by --reneg-* options instead.  This means that OpenVPN will never reuse
> any temporary keys already negotiated if the SSL/TLS connection needs to
> be re-established.  And in fact, that turned out to be pretty good for
> OpenVPN, as the CVE-2009-3555 issue in the SSL/TLS protocol didn't have
> the same impact on OpenVPN as on web servers.
>
> For a few OpenVPN connections, you will most likely not notice much CPU
> hog during the SSL/TLS KEX process.  But if you have some 100 connections
> doing this at the very same time, I expect this be noticed.
>
> And this is also why web servers might not be so sensitive to this, as
> unless you run a high profile SSL site, you probably don't have too many
> simultaneously new SSL/TLS connections being established at the same
> time.  In the moment you have that, the load will rise, and in these
> cases a SSL hardware accelerators will help.  But if such hardware is
> supported by OpenSSL, OpenVPN can also make use of such features.  And
> today, with Intel i5 and i7 CPUs with AES-NI support, this works too
> (--engine aesni).
>
>   
I want to add one remark: key (re)negotiation does NOT depend on the 
encryption cipher, as far as I understand; in any SSL session epheremal 
keys are negotiated based on HMAC values and it makes use of random 
bytes ; especially generating pseudo-random bytes can take some time.

The actual encryption is done using the encryption cipher using the 
epheremal keys. This part shows a nice speedup when using AES-NI capable 
CPUs, but I doubt whether key (re)negotiation is affected.

My advice for scalability would boil down to
- go for clockspeed
- use an AES-NI capable CPU (and corresponding openvpn cipher)
- don't bother with many-core CPUs, dual or quad core is fine
- run multiple openvpn instances, each with their own VPN subnet

The maximum throughput per openvpn process seems to top off at ~ 180 Mbps.

HTH,

JJK

> -----Original Message-----
> From: Jan Just Keijser [mailto:janjust@...]
> Sent: Wednesday, November 23, 2011 11:20 PM
> To: David Sommerseth
> Cc: Jason Haar; openvpn-users@...
> Subject: Re: [Openvpn-users] Scalability
> 
> I want to add one remark: key (re)negotiation does NOT depend on the
> encryption cipher, as far as I understand; in any SSL session epheremal
> keys are negotiated based on HMAC values and it makes use of random
> bytes ; especially generating pseudo-random bytes can take some time.
> 
> The actual encryption is done using the encryption cipher using the
> epheremal keys. This part shows a nice speedup when using AES-NI
> capable
> CPUs, but I doubt whether key (re)negotiation is affected.
> 
> My advice for scalability would boil down to
> - go for clockspeed
> - use an AES-NI capable CPU (and corresponding openvpn cipher)
> - don't bother with many-core CPUs, dual or quad core is fine
> - run multiple openvpn instances, each with their own VPN subnet
> 
> The maximum throughput per openvpn process seems to top off at ~ 180
> Mbps.
> 

I have done tests using UDP that have hit 460Mb using Xeon 5600 CPU's and OpenSSL 1.0.0 using 256 AES. Some performance was also gained by tweaking network buffer settings.

Brad

This email may contain confidential and privileged material for the sole use of the intended recipient. Any review, use, distribution or disclosure by others is strictly prohibited. If you are not the intended recipient (or authorized to receive for the recipient), please contact the sender by reply email and delete all copies of this message.