Menu
▾
▴
openvpn-users
|
From: Olivier S. <oli...@gm...> - 2012-02-21 16:28:36
|
Hi all, we have ~ 160 concurrent users split over 8 OpenVPN processes on a single (8-core) Linux server. The users run mostly terminal server protocols trough the VPN (either Citrix or Nomachine), so lots of small packets but not really high bandwidth (total 10Mbit/s or so). However: we do notice quite a bit of latency once in a while. If we ping the external interface of the VPN, or the internal interface there is normally ~ 4ms difference. But once every minute or so, ping times go up to 2000ms for one or two seconds, and then they drop back to the normal latency. After a bit of testing it seems that all connections to the same OpenVPN process experience that same increase in latency at the same moment. The server cpu load is low. We use OpenVPN 2.1.0, routing (tun devices), udp, and aes-256-cbc. Does anyone have a clue what might be going on, and how to improve this situation? thanks in advance, Olivier |
|
From: Ryan W. <rcw...@gm...> - 2012-02-21 16:33:31
|
Are you running 8 different openvpn processes? On Tue, Feb 21, 2012 at 11:28 AM, Olivier Sessink <oli...@gm...>wrote: > Hi all, > > we have ~ 160 concurrent users split over 8 OpenVPN processes on a > single (8-core) Linux server. The users run mostly terminal server > protocols trough the VPN (either Citrix or Nomachine), so lots of small > packets but not really high bandwidth (total 10Mbit/s or so). > > However: we do notice quite a bit of latency once in a while. If we ping > the external interface of the VPN, or the internal interface there is > normally ~ 4ms difference. But once every minute or so, ping times go up > to 2000ms for one or two seconds, and then they drop back to the normal > latency. > > After a bit of testing it seems that all connections to the same OpenVPN > process experience that same increase in latency at the same moment. The > server cpu load is low. We use OpenVPN 2.1.0, routing (tun devices), > udp, and aes-256-cbc. > > Does anyone have a clue what might be going on, and how to improve this > situation? > > thanks in advance, > > Olivier > > > ------------------------------------------------------------------------------ > Keep Your Developer Skills Current with LearnDevNow! > The most comprehensive online learning library for Microsoft developers > is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, > Metro Style Apps, more. Free future releases when you subscribe now! > http://p.sf.net/sfu/learndevnow-d2d > _______________________________________________ > Openvpn-users mailing list > Ope...@li... > https://lists.sourceforge.net/lists/listinfo/openvpn-users > |
|
From: Olivier S. <oli...@gm...> - 2012-02-21 16:48:35
|
On 02/21/2012 05:33 PM, Ryan Whelan wrote: > Are you running 8 different openvpn processes? yes 8 processes on 8 different udp ports > On Tue, Feb 21, 2012 at 11:28 AM, Olivier Sessink > <oli...@gm... <mailto:oli...@gm...>> wrote: > > Hi all, > > we have ~ 160 concurrent users split over 8 OpenVPN processes on a > single (8-core) Linux server. The users run mostly terminal server > protocols trough the VPN (either Citrix or Nomachine), so lots of small > packets but not really high bandwidth (total 10Mbit/s or so). > > However: we do notice quite a bit of latency once in a while. If we ping > the external interface of the VPN, or the internal interface there is > normally ~ 4ms difference. But once every minute or so, ping times go up > to 2000ms for one or two seconds, and then they drop back to the normal > latency. > > After a bit of testing it seems that all connections to the same OpenVPN > process experience that same increase in latency at the same moment. The > server cpu load is low. We use OpenVPN 2.1.0, routing (tun devices), > udp, and aes-256-cbc. > > Does anyone have a clue what might be going on, and how to improve this > situation? > > thanks in advance, > > Olivier > > ------------------------------------------------------------------------------ > Keep Your Developer Skills Current with LearnDevNow! > The most comprehensive online learning library for Microsoft developers > is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, > Metro Style Apps, more. Free future releases when you subscribe now! > http://p.sf.net/sfu/learndevnow-d2d > _______________________________________________ > Openvpn-users mailing list > Ope...@li... > <mailto:Ope...@li...> > https://lists.sourceforge.net/lists/listinfo/openvpn-users > > |
|
From: Mike T. <mi...@se...> - 2012-02-21 17:02:56
|
On 2/21/2012 11:28 AM, Olivier Sessink wrote: > > Does anyone have a clue what might be going on, and how to improve this > situation? I noticed something similar on an older setup where we used password authentication. When a user would login with the wrong pass, there would be a big spike in latency. It seems something was blocking processing traffic at that point. Does the latency happen at regular intervals ? Whats OpenVPN doing at that time ? A lot of re-keying ? Try and up the logging on the server to see whats happening at the time of the spike. ---Mike -- ------------------- Mike Tancsa, tel +1 519 651 3400 Sentex Communications, mi...@se... Providing Internet services since 1994 www.sentex.net Cambridge, Ontario Canada http://www.tancsa.com/ |
|
From: Olivier S. <oli...@gm...> - 2012-02-21 16:52:20
|
On 02/21/2012 05:43 PM, Mike Tancsa wrote: > On 2/21/2012 11:28 AM, Olivier Sessink wrote: >> >> Does anyone have a clue what might be going on, and how to improve this >> situation? > > > I noticed something similar on an older setup where we used password > authentication. When a user would login with the wrong pass, there > would be a big spike in latency. It seems something was blocking > processing traffic at that point. Does the latency happen at regular > intervals ? Whats OpenVPN doing at that time ? A lot of re-keying ? Try > and up the logging on the server to see whats happening at the time of > the spike. re-keying = key renegotiation ? we set that to 3600 seconds (one hour), so with 15 users on a single openvpn process that should happen once every four minutes. But we see almost every minute those delays (but not always with the same impact, so perhaps key renegotiation is part of the issue). we'll see if we can increase the logging level. However, logging can also become the performance bottleneck ;-) Olivier |
|
From: David S. <ope...@to...> - 2012-02-21 17:52:57
|
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On 21/02/12 17:52, Olivier Sessink wrote:
> On 02/21/2012 05:43 PM, Mike Tancsa wrote:
>> On 2/21/2012 11:28 AM, Olivier Sessink wrote:
>>>
>>> Does anyone have a clue what might be going on, and how to improve
>>> this situation?
>>
>>
>> I noticed something similar on an older setup where we used
>> password authentication. When a user would login with the wrong
>> pass, there would be a big spike in latency. It seems something was
>> blocking processing traffic at that point. Does the latency happen
>> at regular intervals ? Whats OpenVPN doing at that time ? A lot of
>> re-keying ? Try and up the logging on the server to see whats
>> happening at the time of the spike.
>
> re-keying = key renegotiation ?
Yes, correct.
> we set that to 3600 seconds (one hour), so with 15 users on a single
> openvpn process that should happen once every four minutes. But we see
> almost every minute those delays (but not always with the same
> impact, so perhaps key renegotiation is part of the issue).
That's correct if your users connect with such a spread ... but if 5
users connect roughly at the same time, you'll have a higher peak with
those 5 re-keying every hour.
> we'll see if we can increase the logging level. However, logging can
> also become the performance bottleneck ;-)
That might indeed cause an issue on a very loaded server. I'd probably
check if logging using --syslog would decrease that logging load. Then
it's definitely a different process doing the more time consuming disk
writes.
Otherwise, --verb 4 should give you more than enough information about
what's happening - inclusive re-keying. It's not spelled out explicit,
but you'll see lines like the one below at regular intervals:
xxx.xxx.xxx.xxx:yyyy Data Channel Encrypt: Cipher '$cipher' initialized \
with $keylength bit key
xxx.xxx.xxx.xxx:yyyy Data Channel Encrypt: Using $hashlength bit message \
hash '$hashalgorithm' for HMAC authentication
And a similar pair of lines with "Data Channel Decrypt:".
kind regards,
David Sommerseth
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEUEARECAAYFAk9D2eMACgkQDC186MBRfrrjmQCfaidmxxuIvuPYDvjUxsTMuChk
02EAlRXc/SH10LlzdIjcXX2v9J/m0GE=
=P8Tl
-----END PGP SIGNATURE-----
|
|
From: Jason H. <Jas...@tr...> - 2012-02-21 18:27:34
|
Increase your re-keying to 8 hours or more - see if that reduces it. If it does, at least you'll be sure of the root cause. I definitely see openvpn tunnels "freeze" when renegotiation occurs. I think that code needs looking at. We run Cisco IOS IPSec tunnels, they renegotiate keys every 10 hours and we don't see any "freezes". So what does Cisco do that openvpn doesn't? Jason -- Cheers Jason Haar Information Security Manager, Trimble Navigation Ltd. Phone: +1 408 481 8171 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1 |
|
From: David S. <ope...@to...> - 2012-02-22 14:47:18
|
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 21/02/12 19:27, Jason Haar wrote: > Increase your re-keying to 8 hours or more - see if that reduces it. > If it does, at least you'll be sure of the root cause. I definitely > see openvpn tunnels "freeze" when renegotiation occurs. > > I think that code needs looking at. We run Cisco IOS IPSec tunnels, > they renegotiate keys every 10 hours and we don't see any "freezes". > So what does Cisco do that openvpn doesn't? Probably many things. But I suspect the biggest impact might be if it has separate threads to do the key negotiation/exchange (KEX). This way the threads which takes care of the traffic flow will more easily be able to flow freely and not being blocked by the KEX. Another factor might be if it is a complete hardware/software box which also got a built-in RSA/SSL accelerator - which can offload the main CPU from doing much of the heavy lifting the KEX process requires. OpenVPN is a *single threaded* application, which basically runs no faster than what OpenSSL is able to churn - where each client is processed sequentially. This is definitely a drawback with many clients and high activity on the VPN. And if you wonder ... yes, we'd like to make use of some multi-threaded approaches. It just takes time to do that right, and the code needs plenty of cleaning up first before we dare looking into that. But this is really on the roadmap for a future OpenVPN 3 base. But it's not going to happen this year or next year, that's pretty sure :) ... unfortunately. kind regards, David Sommerseth -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk9E/94ACgkQDC186MBRfrpZ/wCffzMRbeg8w90qCdXMaIN9mVDE HMUAn08E6korTijlFgQVFPIlfwyX9RzX =+4lu -----END PGP SIGNATURE----- |
|
From: Michael L. <ml...@ad...> - 2012-02-22 15:47:46
|
On Feb 22, 2012, at 9:46 AM, David Sommerseth wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On 21/02/12 19:27, Jason Haar wrote: >> Increase your re-keying to 8 hours or more - see if that reduces it. >> If it does, at least you'll be sure of the root cause. I definitely >> see openvpn tunnels "freeze" when renegotiation occurs. >> >> I think that code needs looking at. We run Cisco IOS IPSec tunnels, >> they renegotiate keys every 10 hours and we don't see any "freezes". >> So what does Cisco do that openvpn doesn't? > > OpenVPN is a *single threaded* application, which basically runs no > faster than what OpenSSL is able to churn - where each client is > processed sequentially. This is definitely a drawback with many > clients > and high activity on the VPN. > > And if you wonder ... yes, we'd like to make use of some multi- > threaded > approaches. It just takes time to do that right, and the code needs > plenty of cleaning up first before we dare looking into that. But > this > is really on the roadmap for a future OpenVPN 3 base. But it's not > going > to happen this year or next year, that's pretty sure :) ... > unfortunately. > > > kind regards, > > David Sommerseth <disclaimer> I am not a C programmer, nor have I ever programmed threaded applications </disclaimer> As an interim solution to a fully threaded application, would it work to put just the key regeneration routine in a separate thread? In theory, communications wouldn't be impacted while the key regen ran, and the development team would have the time they need to make the entire application thread safe while getting the benefits of a two thread application. Just a thought, --Michael |
|
From: Les M. <les...@gm...> - 2012-02-22 16:10:03
|
On Wed, Feb 22, 2012 at 9:47 AM, Michael Leahy
<ml...@ad...> wrote:
>
> <disclaimer>
> I am not a C programmer, nor have I ever programmed threaded
> applications
> </disclaimer>
>
> As an interim solution to a fully threaded application, would it work
> to put just the key regeneration routine in a separate thread? In
> theory, communications wouldn't be impacted while the key regen ran,
> and the development team would have the time they need to make the
> entire application thread safe while getting the benefits of a two
> thread application.
>
Usually it takes about a decade to shake the bugs out when you try to
go from single-threaded to threaded apps or libs because of all the
shared state. It might be much safer to keep a separate process
running to do the time-consuming parts, chatting with it over a socket
while other events continue.
--
Les Mikesell
les...@gm...
|
|
From: David S. <ope...@to...> - 2012-02-22 16:29:10
|
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 22/02/12 16:47, Michael Leahy wrote: > On Feb 22, 2012, at 9:46 AM, David Sommerseth wrote: > >> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 >> >> On 21/02/12 19:27, Jason Haar wrote: >>> Increase your re-keying to 8 hours or more - see if that reduces >>> it. If it does, at least you'll be sure of the root cause. I >>> definitely see openvpn tunnels "freeze" when renegotiation >>> occurs. >>> >>> I think that code needs looking at. We run Cisco IOS IPSec >>> tunnels, they renegotiate keys every 10 hours and we don't see >>> any "freezes". So what does Cisco do that openvpn doesn't? >> >> OpenVPN is a *single threaded* application, which basically runs no >> faster than what OpenSSL is able to churn - where each client is >> processed sequentially. This is definitely a drawback with many >> clients and high activity on the VPN. >> >> And if you wonder ... yes, we'd like to make use of some >> multi-threaded approaches. It just takes time to do that right, >> and the code needs plenty of cleaning up first before we dare >> looking into that. But this is really on the roadmap for a future >> OpenVPN 3 base. But it's not going to happen this year or next >> year, that's pretty sure :) ... unfortunately. >> >> >> kind regards, >> >> David Sommerseth > > <disclaimer> I am not a C programmer, nor have I ever programmed > threaded applications </disclaimer> > > As an interim solution to a fully threaded application, would it work > to put just the key regeneration routine in a separate thread? In > theory, communications wouldn't be impacted while the key regen ran, > and the development team would have the time they need to make the > entire application thread safe while getting the benefits of a two > thread application. Yes, and that has been attempted. But that implementation was not really completed. And as it had stayed in the source tree for a very long time without being touched, we decided to kill that code sometime before the 2.2 release. Not because the idea was bad, but because implementing it in a good way which gives real benefits in OpenVPN is difficult. And the current state of that code was not even compilable. At that time, it was decided to rather look at this in conjunction with the OpenVPN 3 roadmap, which will have a better multi-threaded design from the ground. OpenVPN is quite remarkable in how it has developed, where it is still a single threaded application which really performs very well under most configurations. And that is still 10 years after the first release. And the first releases where designed to have one "server" process per client, in a straight site-to-site setup. The support for multiple clients in a single server process was first added in the 2.0 release, and where it still stayed single threaded. And the advantage of that is that the locking mechanisms which multiple threads needs to use, to avoid conflicts on shared resources, is a problem we've never really had in OpenVPN. However, the disadvantage is that "everyone" stands in a single queue, waiting to be served sequentially. kind regards, David Sommerseth -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk9FF74ACgkQDC186MBRfrrqFwCfX+vCa7upyiMMpylHCRUW6ci0 q/UAoIU1G46JWN+f6nFfSSx0s46pMwtE =0vAs -----END PGP SIGNATURE----- |
|
From: Mike T. <mi...@se...> - 2012-02-23 00:44:48
|
On 2/22/2012 10:47 AM, Michael Leahy wrote: > On Feb 22, 2012, at 9:46 AM, David Sommerseth wrote: >> On 21/02/12 19:27, Jason Haar wrote: >>> Increase your re-keying to 8 hours or more - see if that reduces it. >>> If it does, at least you'll be sure of the root cause. I definitely >>> see openvpn tunnels "freeze" when renegotiation occurs. > > As an interim solution to a fully threaded application, would it work > to put just the key regeneration routine in a separate thread? In Depending on your setup, another option might be just to run a few more instances of OpenVPN on different ports. If you want to share the load among 4 cores, start up 4 separate daemons. The clients can be configured to round-robin through them. ---Mike -- ------------------- Mike Tancsa, tel +1 519 651 3400 Sentex Communications, mi...@se... Providing Internet services since 1994 www.sentex.net Cambridge, Ontario Canada http://www.tancsa.com/ |
|
From: <J.W...@mi...> - 2012-03-07 15:30:06
|
See below: -----Original Message----- From: David Sommerseth [mailto:ope...@to...] Sent: Wednesday, February 22, 2012 3:47 PM To: Jason Haar Cc: ope...@li... Subject: Re: [Openvpn-users] latency issues on larger scale OpenVPN setup -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Probably many things. But I suspect the biggest impact might be if it has separate threads to do the key negotiation/exchange (KEX). This way the threads which takes care of the traffic flow will more easily be able to flow freely and not being blocked by the KEX. Another factor might be if it is a complete hardware/software box which also got a built-in RSA/SSL accelerator - which can offload the main CPU from doing much of the heavy lifting the KEX process requires. OpenVPN is a *single threaded* application, which basically runs no faster than what OpenSSL is able to churn - where each client is processed sequentially. This is definitely a drawback with many clients and high activity on the VPN. And if you wonder ... yes, we'd like to make use of some multi-threaded approaches. It just takes time to do that right, and the code needs plenty of cleaning up first before we dare looking into that. But this is really on the roadmap for a future OpenVPN 3 base. But it's not going to happen this year or next year, that's pretty sure :) ... unfortunately. kind regards, David Sommerseth ------------------------------------------------------------------------------ Hi David, What puzzles me, is that is if re-keying is blocking everything (all users) within that server-process, I would expect that during that time, those processes would get 100% CPU. But they don't. I've seen other processes (like mysql) getting a cpu-burst, but all the vpn-processes are on average between 5 and 15 % cpu. I got a hifn crypto-processors, but never installed, as I got the impression that modern CPU's were capable enough. Btw, in our box we got dual quad-core Intel E5440 Hans ______________________________________________________________________ Dit bericht kan informatie bevatten die niet voor u is bestemd. Indien u niet de geadresseerde bent of dit bericht abusievelijk aan u is toegezonden, wordt u verzocht dat aan de afzender te melden en het bericht te verwijderen. De Staat aanvaardt geen aansprakelijkheid voor schade, van welke aard ook, die verband houdt met risico's verbonden aan het elektronisch verzenden van berichten. This message may contain information that is not intended for you. If you are not the addressee or if this message was sent to you by mistake, you are requested to inform the sender and delete the message. The State accepts no liability for damage of any kind resulting from the risks inherent in the electronic transmission of messages. |
|
From: David S. <ope...@to...> - 2012-03-07 16:22:01
|
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 07/03/12 16:29, J.W...@mi... wrote: > See below: > > -----Original Message----- From: David Sommerseth > [mailto:ope...@to...] Sent: Wednesday, February 22, > 2012 3:47 PM To: Jason Haar Cc: ope...@li... > Subject: Re: [Openvpn-users] latency issues on larger scale OpenVPN > setup > > -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 > > > Probably many things. But I suspect the biggest impact might be if > it has separate threads to do the key negotiation/exchange (KEX). > This way the threads which takes care of the traffic flow will more > easily be able to flow freely and not being blocked by the KEX. > > Another factor might be if it is a complete hardware/software box > which also got a built-in RSA/SSL accelerator - which can offload the > main CPU from doing much of the heavy lifting the KEX process > requires. > > OpenVPN is a *single threaded* application, which basically runs no > faster than what OpenSSL is able to churn - where each client is > processed sequentially. This is definitely a drawback with many > clients and high activity on the VPN. > > And if you wonder ... yes, we'd like to make use of some > multi-threaded approaches. It just takes time to do that right, and > the code needs plenty of cleaning up first before we dare looking into > that. But this is really on the roadmap for a future OpenVPN 3 base. > But it's not going to happen this year or next year, that's pretty > sure :) ... unfortunately. > > > kind regards, > > David Sommerseth > ------------------------------------------------------------------------------ > > Hi David, > > What puzzles me, is that is if re-keying is blocking everything (all > users) within that server-process, I would expect that during that > time, those processes would get 100% CPU. But they don't. I've seen > other processes (like mysql) getting a cpu-burst, but all the > vpn-processes are on average between 5 and 15 % cpu. I got a hifn > crypto-processors, but never installed, as I got the impression that > modern CPU's were capable enough. Btw, in our box we got dual > quad-core Intel E5440 I've not put my head deep enough into that part of the OpenVPN code, to see how exactly this KEX process is tackled here. But from what I know, the KEX process do really lock other connections. During the KEX process, there is a bi-directional communication between client and server as well. So it isn't only related to the server side. If the server side needs to wait for the client to respond, you'll have a low-peak with a block. Other things which I didn't mention last time also can halt the other connections, is authentication and external certificate checks, used via the the different script hooks and/or the --plugin interface. If that requires communication with external services, you'll have another blocker. The only exception here is if a --plugin is written with 'deferred authentication' support, but that is only available with username/password authentication. And even though your box is a 2x quad-core box (8 cores in total), one single OpenVPN instance will still only run on one of these cores. To make use of the other cores, you will for now need to spawn seven more OpenVPN processes, and pin them to each CPU core (f.ex via taskset). But when multi-thread supports comes in OpenVPN, one single process can make better use of the other cores as well. kind regards, David Sommerseth -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk9XivwACgkQDC186MBRfrosUwCdE2gidn2etut6nD0aRxa3yrv5 V8IAnRKU62ThUzvrDYQIQs7sTRaDUXbP =7GoY -----END PGP SIGNATURE----- |
|
From: <J.W...@mi...> - 2012-03-08 17:23:28
|
-----Original Message----- From: David Sommerseth [mailto:ope...@to...] Sent: Wednesday, March 07, 2012 5:22 PM To: Witvliet, J, CDC/IV/DCOPS/I&S/HIN Cc: ope...@li... Subject: Re: [Openvpn-users] latency issues on larger scale OpenVPN setup Hi David, > > What puzzles me, is that is if re-keying is blocking everything (all > users) within that server-process, I would expect that during that > time, those processes would get 100% CPU. But they don't. I've seen > other processes (like mysql) getting a cpu-burst, but all the > vpn-processes are on average between 5 and 15 % cpu. I got a hifn > crypto-processors, but never installed, as I got the impression that > modern CPU's were capable enough. Btw, in our box we got dual > quad-core Intel E5440 I've not put my head deep enough into that part of the OpenVPN code, to see how exactly this KEX process is tackled here. But from what I know, the KEX process do really lock other connections. During the KEX process, there is a bi-directional communication between client and server as well. So it isn't only related to the server side. If the server side needs to wait for the client to respond, you'll have a low-peak with a block. Other things which I didn't mention last time also can halt the other connections, is authentication and external certificate checks, used via the the different script hooks and/or the --plugin interface. If that requires communication with external services, you'll have another blocker. The only exception here is if a --plugin is written with 'deferred authentication' support, but that is only available with username/password authentication. And even though your box is a 2x quad-core box (8 cores in total), one single OpenVPN instance will still only run on one of these cores. To make use of the other cores, you will for now need to spawn seven more OpenVPN processes, and pin them to each CPU core (f.ex via taskset). But when multi-thread supports comes in OpenVPN, one single process can make better use of the other cores as well. kind regards, David Sommerseth _______________________________________________ Hi David, Yes, we noticed that any delay in tls_verify, client_connect or client_disconnect also effects any other vpn-session. We already had multiple vpn-services running, and each of those seven processes had over fifty connections. I understand that during the early design phase, it was decided not to respawn child-processes, due to security reasons. Absolutely no problems with that. But it is strange that within one vpn-process, the re-keying/set-up/crl-check/etc of one connection could influence others. Not? Only way to avoid such mutual influence looks like to give every tunnel its own openvpn-process. But I'll guess I run into other problems when trying to startup thousand or more processes..... Kind Regards, Hans ______________________________________________________________________ Dit bericht kan informatie bevatten die niet voor u is bestemd. Indien u niet de geadresseerde bent of dit bericht abusievelijk aan u is toegezonden, wordt u verzocht dat aan de afzender te melden en het bericht te verwijderen. De Staat aanvaardt geen aansprakelijkheid voor schade, van welke aard ook, die verband houdt met risico's verbonden aan het elektronisch verzenden van berichten. This message may contain information that is not intended for you. If you are not the addressee or if this message was sent to you by mistake, you are requested to inform the sender and delete the message. The State accepts no liability for damage of any kind resulting from the risks inherent in the electronic transmission of messages. |
|
From: Olivier S. <oli...@gm...> - 2012-03-08 21:45:45
|
On 03/08/2012 06:23 PM, J.W...@mi... wrote: > > Hi David, Yes, we noticed that any delay in tls_verify, > client_connect or client_disconnect also effects any other > vpn-session. and those can be waiting for I/O. That's why you don't see the 100% cpu load, those scripts are just waiting for I/O and stalling the openvpn process. Olivier |
|
From: David S. <ope...@to...> - 2012-03-08 22:20:57
|
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 08/03/12 18:23, J.W...@mi... wrote: [...snip...] > I understand that during the early design phase, it was decided not > to respawn child-processes, due to security reasons. Absolutely no > problems with that. But it is strange that within one vpn-process, the > re-keying/set-up/crl-check/etc of one connection could influence > others. Not? To fully understand this, you need to know a little bit more of the OpenVPN history. The v1.x series of OpenVPN, AFAIR, didn't support more than one single connection. There were no multi-client support which we're used to since v2.0. OpenVPN was initially designed as a site-to-site VPN, where you would configure one daemon for each site you wanted to hook up. And in that perspective, the current design makes a lot of sense. However, when you start a cool project, new features are requested. And for some reason, it was decided to extended the current code base to support multiple clients, without multi-thread support. At that time, servers with multiple CPUs were not that common among most use cases where OpenVPN was deployed, so it wouldn't really make that much difference. As it would have to be the same CPU core doing all the work anyway. But time has changed since that time too ... and now there's even laptops with 4 and 8 cores. In today's world, not having threading support is baroque. > Only way to avoid such mutual influence looks like to give every > tunnel its own openvpn-process. But I'll guess I run into other > problems when trying to startup thousand or more processes..... That would be another pile of issues to solve, for sure. But it's closer to the original design of OpenVPN than what you probably would have guessed ;-) kind regards, David Sommerseth -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk9ZMLgACgkQDC186MBRfrqXLwCcDIkzbPqFqiEHaO03eSfXtqwv VyUAn2ysLzVqeFZ8Fju4BDly2gKQLOrD =sLXj -----END PGP SIGNATURE----- |
|
From: Les M. <les...@gm...> - 2012-03-08 17:48:04
|
On Thu, Mar 8, 2012 at 11:23 AM, <J.W...@mi...> wrote:
>
> I understand that during the early design phase, it was decided not to respawn child-processes, due to security reasons.
> Absolutely no problems with that. But it is strange that within one vpn-process, the re-keying/set-up/crl-check/etc of one connection could influence others. Not?
>
> Only way to avoid such mutual influence looks like to give every tunnel its own openvpn-process.
> But I'll guess I run into other problems when trying to startup thousand or more processes.....
>
A few thousand processes shouldn't be a big deal. Managing the
configs with separate ports for each connection might be. Without
really understanding the code, I'd guess a separate process to handle
re-keying that can chat over a local socket in the main event loop
would be the easiest way to avoid making other connections wait.
--
Les Mikesell
les...@gm...
|
|
From: Jason H. <Jas...@tr...> - 2012-03-08 20:43:05
|
...setting your re-keying to something huge helps too. I seriously doubt many of us need to be concerned with the risk of rotating our session keys - at all... (see --reneg-sec) On 09/03/12 06:47, Les Mikesell wrote: > On Thu, Mar 8, 2012 at 11:23 AM, <J.W...@mi...> wrote: >> I understand that during the early design phase, it was decided not to respawn child-processes, due to security reasons. >> Absolutely no problems with that. But it is strange that within one vpn-process, the re-keying/set-up/crl-check/etc of one connection could influence others. Not? >> >> Only way to avoid such mutual influence looks like to give every tunnel its own openvpn-process. >> But I'll guess I run into other problems when trying to startup thousand or more processes..... >> > A few thousand processes shouldn't be a big deal. Managing the > configs with separate ports for each connection might be. Without > really understanding the code, I'd guess a separate process to handle > re-keying that can chat over a local socket in the main event loop > would be the easiest way to avoid making other connections wait. > -- Cheers Jason Haar Information Security Manager, Trimble Navigation Ltd. Phone: +1 408 481 8171 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1 |
|
From: David S. <ope...@to...> - 2012-03-08 22:25:27
|
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 08/03/12 18:47, Les Mikesell wrote: > On Thu, Mar 8, 2012 at 11:23 AM, <J.W...@mi...> wrote: >> >> I understand that during the early design phase, it was decided not >> to respawn child-processes, due to security reasons. Absolutely no >> problems with that. But it is strange that within one vpn-process, >> the re-keying/set-up/crl-check/etc of one connection could influence >> others. Not? >> >> Only way to avoid such mutual influence looks like to give every >> tunnel its own openvpn-process. But I'll guess I run into other >> problems when trying to startup thousand or more processes..... >> > > A few thousand processes shouldn't be a big deal. Managing the > configs with separate ports for each connection might be. Without > really understanding the code, I'd guess a separate process to handle > re-keying that can chat over a local socket in the main event loop > would be the easiest way to avoid making other connections wait. Configs shouldn't be that hard to solve. You can include config files within a config ... just a very stupid example ... - ---config-ssl.conf--------- tls-server ca keys/ca.pem cert keys/servercert.pem key keys/serverkey.pem - --------------------------- - ---config-port1194.conf---- config config-ssl.conf port 1194 mode server (etc, etc) - --------------------------- Remember that all options given on the command line can be used in the config file. No exception, it's the same parser code doing both. You just remove the '--' from the command line when putting it in the config file. And if you have the same options several places, it's the last parsed one which counts, and I believe the command line is the last to be parsed. kind regards, David Sommerseth -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk9ZMccACgkQDC186MBRfro1CACgj3W0dVm8NJ29FSuXb0Pq/6YQ UhIAoICSdmvBnpMteBYOimI/QXe53qqJ =2mRz -----END PGP SIGNATURE----- |
|
From: David S. <ope...@to...> - 2012-03-08 22:13:56
|
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 22/02/12 16:47, Michael Leahy wrote: > On Feb 22, 2012, at 9:46 AM, David Sommerseth wrote: >> >> OpenVPN is a *single threaded* application, which basically runs no >> faster than what OpenSSL is able to churn - where each client is >> processed sequentially. This is definitely a drawback with many >> clients and high activity on the VPN. >> >> And if you wonder ... yes, we'd like to make use of some >> multi-threaded approaches. It just takes time to do that right, and >> the code needs plenty of cleaning up first before we dare looking >> into that. But this is really on the roadmap for a future OpenVPN 3 >> base. But it's not going to happen this year or next year, that's >> pretty sure :) ... unfortunately. >> > <disclaimer> I am not a C programmer, nor have I ever programmed > threaded applications </disclaimer> > > As an interim solution to a fully threaded application, would it work > to put just the key regeneration routine in a separate thread? In > theory, communications wouldn't be impacted while the key regen ran, > and the development team would have the time they need to make the > entire application thread safe while getting the benefits of a two > thread application. > > Just a thought, It sure is a good thought, and in many applications, this would just work out fine. However, we had some code paths which tried to do so in OpenVPN, but the implementation was never completed - in fact, it didn't even compile(!) So after laying dead for some years, we ripped it out. (Somehow, I feel I've mentioned this in an earlier thread too). The challenge with OpenVPN is that redesigning a single-threaded application to become a multi-threaded one is not as easy as it often sounds like. And you would actually be able to manage pretty much the same result as if you would just do it in a single thread. If OpenVPN would just defer the waiting while re-keying, taking care of some other connections in the mean time, this would solve this lag. And without having looked too much at the previous attempt, I believe this was what it was trying to do. But this approach makes it tricky, as the re-keying might be prune to fail if the further RSA process is postponed for too long. And this could happen in a large site, with many active VPN clients pushing data. However, the challenge in that dual-thread approach is how OpenVPN the connections are handled internally, with RSA keying happening in a separate thread. The socket and event handling code would have to be modified largely. This so that the event handler could dispatch a client ready with keying material to the RSA thread, and then the socket code would need to get allowance to read the data from the client - so that it wouldn't interrupting other connections in the middle of a transfer. So several mutex locks would need to be implemented in the proper places. So with all this work needed to be done for just this extra single thread, it's far better to do a major overhaul of the whole event and socket code all together. This way it could scale even better to more threads if CPU cores are available. And also considering quad and octa core CPUs getting more and more common, aiming for a dual-thread solution would be rather baroque - even before we would release such an implementation. kind regards, David Sommerseth -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk9ZLxQACgkQDC186MBRfrp4FgCeMpjjHE4T58Y7fJnLiaMhAWQQ RNcAn0suIua+HAEKpYrW70hSORoEeTBe =tBoO -----END PGP SIGNATURE----- |
|
From: Samuli S. <sa...@op...> - 2012-03-09 08:52:46
|
> The challenge with OpenVPN is that redesigning a single-threaded > application to become a multi-threaded one is not as easy as it often > sounds like. And you would actually be able to manage pretty much the > same result as if you would just do it in a single thread. If OpenVPN > would just defer the waiting while re-keying, taking care of some other > connections in the mean time, this would solve this lag. And without > having looked too much at the previous attempt, I believe this was what > it was trying to do. But this approach makes it tricky, as the re-keying > might be prune to fail if the further RSA process is postponed for too > long. And this could happen in a large site, with many active VPN > clients pushing data. > Let me play the devil's advocate here.. what if the current approach - single-threaded but multiple processes - makes more sense than going multi-threaded? Threads are lighter than processes, but also make the code more complex (and error-prone). Do we get any other benefits from threads besides better utilisation of multiple processor cores? -- Samuli Seppänen Community Manager OpenVPN Technologies, Inc irc freenode net: mattock |
|
From: Samuli S. <sa...@op...> - 2012-03-09 09:12:28
|
>> The challenge with OpenVPN is that redesigning a single-threaded >> application to become a multi-threaded one is not as easy as it often >> sounds like. And you would actually be able to manage pretty much the >> same result as if you would just do it in a single thread. If OpenVPN >> would just defer the waiting while re-keying, taking care of some other >> connections in the mean time, this would solve this lag. And without >> having looked too much at the previous attempt, I believe this was what >> it was trying to do. But this approach makes it tricky, as the re-keying >> might be prune to fail if the further RSA process is postponed for too >> long. And this could happen in a large site, with many active VPN >> clients pushing data. >> > Let me play the devil's advocate here.. what if the current approach - > single-threaded but multiple processes - makes more sense than going > multi-threaded? Threads are lighter than processes, but also make the > code more complex (and error-prone). Do we get any other benefits from > threads besides better utilisation of multiple processor cores? > I might add that use of threads makes sense if single client connections waiting for I/O block other clients too often (topic of this thread). In this case isolating each connection is useful, and threads are quicker and less computing-intensive to create than processes. Besides the client initialisation/key renegotiation are there other places where the whole OpenVPN process could get blocked by a single client? -- Samuli Seppänen Community Manager OpenVPN Technologies, Inc irc freenode net: mattock |
|
From: Brad D. (Ericsson) <Bra...@cl...> - 2012-02-23 20:03:22
|
> -----Original Message----- > From: Olivier Sessink [mailto:oli...@gm...] > Sent: Tuesday, February 21, 2012 8:28 AM > To: ope...@li... > Subject: [Openvpn-users] latency issues on larger scale OpenVPN setup > > Hi all, > > we have ~ 160 concurrent users split over 8 OpenVPN processes on a > single (8-core) Linux server. The users run mostly terminal server > protocols trough the VPN (either Citrix or Nomachine), so lots of small > packets but not really high bandwidth (total 10Mbit/s or so). > > However: we do notice quite a bit of latency once in a while. If we > ping the external interface of the VPN, or the internal interface there > is normally ~ 4ms difference. But once every minute or so, ping times > go up to 2000ms for one or two seconds, and then they drop back to the > normal latency. > > After a bit of testing it seems that all connections to the same > OpenVPN process experience that same increase in latency at the same > moment. The server cpu load is low. We use OpenVPN 2.1.0, routing (tun > devices), udp, and aes-256-cbc. > > Does anyone have a clue what might be going on, and how to improve this > situation? > > thanks in advance, > > Olivier This is interesting as I only run 2 processes per server and have over 1000 users per process with no issues on a dual dual-core server. This has been running for about 4 years now. Couple questions. What Linux distro? What OpenSSL version are you running? What is the model of processor in your VPN server? I ask this because the newer Intel procs have AES hardware in them. And using OpenSSL 1.0.0 or newer takes advantage of this. Most Linux distro's still use 0.9.8. However even without the hardware AES just upgrading OpenSSL did increase throughput dramatically. In my lab testing I saw around 300Mbps on just a CentOS 5.2 x86_64 install. I upgraded to OpenSSL 1.0.0 and saw 360Mbps. Then on my hardware AES I saw well over 460Mbps. This was using a VM instance for the AES hardware testing. So I assume I could of gotten plenty more on full hardware. Brad This email may contain confidential and privileged material for the sole use of the intended recipient. Any review, use, distribution or disclosure by others is strictly prohibited. If you are not the intended recipient (or authorized to receive for the recipient), please contact the sender by reply email and delete all copies of this message. |
|
From: Jan J. K. <ja...@ni...> - 2012-02-23 21:29:18
|
Hi Brad, Brad Dameron (Ericsson) wrote: >> -----Original Message----- >> From: Olivier Sessink [mailto:oli...@gm...] >> Sent: Tuesday, February 21, 2012 8:28 AM >> To: ope...@li... >> Subject: [Openvpn-users] latency issues on larger scale OpenVPN setup >> >> Hi all, >> >> we have ~ 160 concurrent users split over 8 OpenVPN processes on a >> single (8-core) Linux server. The users run mostly terminal server >> protocols trough the VPN (either Citrix or Nomachine), so lots of small >> packets but not really high bandwidth (total 10Mbit/s or so). >> >> However: we do notice quite a bit of latency once in a while. If we >> ping the external interface of the VPN, or the internal interface there >> is normally ~ 4ms difference. But once every minute or so, ping times >> go up to 2000ms for one or two seconds, and then they drop back to the >> normal latency. >> >> After a bit of testing it seems that all connections to the same >> OpenVPN process experience that same increase in latency at the same >> moment. The server cpu load is low. We use OpenVPN 2.1.0, routing (tun >> devices), udp, and aes-256-cbc. >> >> Does anyone have a clue what might be going on, and how to improve this >> situation? >> >> thanks in advance, >> >> Olivier >> > > This is interesting as I only run 2 processes per server and have over 1000 users per process with no issues on a dual dual-core server. This has been running for about 4 years now. Couple questions. What Linux distro? What OpenSSL version are you running? What is the model of processor in your VPN server? I ask this because the newer Intel procs have AES hardware in them. And using OpenSSL 1.0.0 or newer takes advantage of this. Most Linux distro's still use 0.9.8. However even without the hardware AES just upgrading OpenSSL did increase throughput dramatically. In my lab testing I saw around 300Mbps on just a CentOS 5.2 x86_64 install. I upgraded to OpenSSL 1.0.0 and saw 360Mbps. Then on my hardware AES I saw well over 460Mbps. This was using a VM instance for the AES hardware testing. So I assume I could of gotten plenty more on full hardware. > that's a EL5/CentOS 5 specific issue: RedHat managed to break the performance of OpenSSL on EL5 ; recompiling a stock OpenSSL 0.9.8 on EL5 shows the same performance improvements... having said that, the AESNI stuff certainly helps OpenVPN on the right hardware. HTH, JJK |
