From: Brian C. <B.C...@po...> - 2006-03-29 09:45:22
|
I am trying to do some load-testing between a Linux box and Cisco IOS, for L2TP over IPSEC. I'm using CentOS 4.2 (kernel 2.6.9-34.EL), ipsec-tools CVS HEAD plus rp-l2tp 0.4. I have a script which triggers off a bunch of l2tpd's, each bound to a separate local port, with separate SPD entries. After a certain number of tunnels, I start to get these errors. For example, if I try to set up 70 tunnels, 66 of them work but I get four messages of the form Mar 29 10:24:02 candlerb racoon: ERROR: failed to recv from pfkey (Resource temporarily unavailable) Mar 29 10:24:02 candlerb racoon: ERROR: failed to recv from pfkey (Resource temporarily unavailable) Mar 29 10:24:02 candlerb racoon: ERROR: failed to recv from pfkey (Resource temporarily unavailable) Mar 29 10:24:02 candlerb racoon: ERROR: failed to recv from pfkey (Resource temporarily unavailable) ... Mar 29 10:24:32 candlerb racoon: INFO: IPsec-SA expired: ESP/Transport X.X.X.X[0]->Y.Y.Y.Y[0] spi=26461726(0x193c61e) Mar 29 10:24:32 candlerb racoon: ERROR: X.X.X.X give up to get IPsec-SA due to time up to wait. Mar 29 10:24:32 candlerb last message repeated 3 times Mar 29 10:24:32 candlerb racoon: INFO: IPsec-SA expired: ESP/Transport X.X.X.X[0]->Y.Y.Y.Y[0] spi=266244205(0xfde906d) Mar 29 10:24:32 candlerb racoon: INFO: IPsec-SA expired: ESP/Transport X.X.X.X[0]->Y.Y.Y.Y[0] spi=104708991(0x63dbb7f) Mar 29 10:24:32 candlerb racoon: INFO: IPsec-SA expired: ESP/Transport X.X.X.X[0]->Y.Y.Y.Y[0] spi=111858690(0x6aad402) and only 66 pppd interfaces are created. My question is, am I hitting some fundamental scaling limitation of IPSEC under Linux, or is there something I can do to increase this? Possibly related: with 70 pairs of IPSEC policies I am able to dump the SPD an SAD successfully. # echo "spddump;" | setkey -c # echo "dump;" | setkey -c However, when I tried a larger number of tunnels (250) this fails: # echo "spddump;" | setkey -c ... X.X.X.X[1701] Y.Y.Y.Y[10011] udp fwd prio def ipsec esp/transport//require created: Mar 29 10:36:50 2006 lastused: lifetime: 0(s) validtime: 0(s) spid=790930 seq=242 pid=30570 refcnt=1 X.X.X.X[1701] Y.Y.Y.Y[10012] udp fwd prio def ipsec esp/transport//require created: Mar 29 10:36:50 2006 lastused: lifetime: 0(s) validtime: 0(s) spid=790954 seq=241 pid=30570 refcnt=1 recv: Resource temporarily unavailable # Many thanks, Brian. |
From: jamal <ha...@cy...> - 2006-03-29 11:33:19
|
thats likely pfkey biting you in the ass. Not only does racoon/pfkey use a small socket buffer, in general it is not very scalable when you have a large number of SAs or SPDs (the problem being pfkey and trying to retrieve large # of objects from the kernel). So you could try to increase the hard-coded socket buffers and stagger your creation of the l2tp tunnels or simply use xfrm/netlink instead of setkey wherever you can. In my opinion as a command line utility ipxfrm is a lot more intuitive. example: "ip xfrm policy list" to dump. See if this causes you the same issues with 250 polices. cheers, jamal On Wed, 2006-29-03 at 10:45 +0100, Brian Candler wrote: [..] > My question is, am I hitting some fundamental scaling limitation of IPSEC > under Linux, or is there something I can do to increase this? > > Possibly related: with 70 pairs of IPSEC policies I am able to dump the SPD > an SAD successfully. > > # echo "spddump;" | setkey -c > # echo "dump;" | setkey -c > > However, when I tried a larger number of tunnels (250) this fails: |
From: Brian C. <B.C...@po...> - 2006-03-29 12:12:22
|
On Wed, Mar 29, 2006 at 06:31:59AM -0500, jamal wrote: > thats likely pfkey biting you in the ass. > Not only does racoon/pfkey use a small socket buffer, in general it is > not very scalable when you have a large number of SAs or SPDs (the > problem being pfkey and trying to retrieve large # of objects from the > kernel). So you could try to increase the hard-coded socket buffers and > stagger your creation of the l2tp tunnels or simply use xfrm/netlink > instead of setkey wherever you can. In my opinion as a command line > utility ipxfrm is a lot more intuitive. > example: > "ip xfrm policy list" to dump. See if this causes you the same issues > with 250 polices. Thank you - that at least lets me see the 250 sets of policies installed. However there still seem to be problems with racoon communicating with the kernel. By retrying some tunnels I was able to get up to 178, but after this I start getting these errors: Mar 29 12:43:13 candlerb racoon: ERROR: no policy found: id:811033. Mar 29 12:43:13 candlerb racoon: ERROR: no policy found: id:811057. (that's when trying to start tunnels 179 and 180) Where are the hard-coded socket buffers you're talking about defined? In racoon/pfkey.c, function pk_recv(), it appears to use a dynamically resized buffer. Is this limit something in the kernel side? Cheers, Brian. |
From: jamal <ha...@cy...> - 2006-03-29 12:48:07
|
On Wed, 2006-29-03 at 13:12 +0100, Brian Candler wrote: > Thank you - that at least lets me see the 250 sets of policies installed. > > However there still seem to be problems with racoon communicating with the > kernel. By retrying some tunnels I was able to get up to 178, but after this > I start getting these errors: > > Mar 29 12:43:13 candlerb racoon: ERROR: no policy found: id:811033. > Mar 29 12:43:13 candlerb racoon: ERROR: no policy found: id:811057. > > (that's when trying to start tunnels 179 and 180) > The way i read that is it is caused by acquires coming from the kernel with those two policy ids and racoon being unable to find the policies. It is as if the kernel has these policies but not racoon which would be strange if racoon installed them to begin with. Does ipxfrm show those policy ids? > Where are the hard-coded socket buffers you're talking about defined? In > racoon/pfkey.c, function pk_recv(), it appears to use a dynamically resized > buffer. Is this limit something in the kernel side? > Look at src/libipsec/pfkey.c::pfkey_open() its set to about 128K at the moment. cheers, jamal |
From: Brian C. <B.C...@po...> - 2006-03-29 13:51:57
|
On Wed, Mar 29, 2006 at 07:47:56AM -0500, jamal wrote: > Look at src/libipsec/pfkey.c::pfkey_open() > its set to about 128K at the moment. I have now changed that to 1024K. Thanks. > > Mar 29 12:43:13 candlerb racoon: ERROR: no policy found: id:811033. > > Mar 29 12:43:13 candlerb racoon: ERROR: no policy found: id:811057. > > > > (that's when trying to start tunnels 179 and 180) > > > > The way i read that is it is caused by acquires coming from the kernel > with those two policy ids and racoon being unable to find the policies. > It is as if the kernel has these policies but not racoon which would be > strange if racoon installed them to begin with. > Does ipxfrm show those policy ids? Right, I've tested this again. Trying to bring up 250 tunnels, 90 succeeded at first attempt. I see one error "failed to recv from pfkey (Resource temporarily unavailable)", as well as 34 instances like Mar 29 14:29:08 candlerb racoon: ERROR: no policy found: id:828457 Mar 29 14:29:08 candlerb racoon: ERROR: no policy found: id:828481 However I don't know how to get policy ids from ip xfrm; the output looks like this # ip xfrm policy list src X.X.X.X/32 dst Y.Y.Y.Y/32 proto udp sport 1701 dport 10000 dir in priority 2147483648 tmpl src 0.0.0.0 dst 0.0.0.0 proto ipv6-crypt spi 0x00000000 reqid 22047 mode transport src X.X.X.X/32 dst Y.Y.Y.Y/32 proto udp sport 1701 dport 10001 dir in priority 2147483648 tmpl src 0.0.0.0 dst 0.0.0.0 proto ipv6-crypt spi 0x00000000 reqid 22049 mode transport ... etc I don't see a 'policy id' there. I can see it with setkey: # echo "spddump;" | setkey -c | less X.X.X.X[1701] Y.Y.Y.Y[10000] udp in prio def ipsec esp/transport//unique#22047 created: Mar 29 14:28:34 2006 lastused: Mar 29 14:35:44 2006 lifetime: 0(s) validtime: 0(s) spid=822512 seq=753 pid=29037 refcnt=2 X.X.X.X[1701] Y.Y.Y.Y[10001] udp in prio def ipsec esp/transport//unique#22049 created: Mar 29 14:28:34 2006 lastused: Mar 29 14:35:44 2006 lifetime: 0(s) validtime: 0(s) spid=822536 seq=752 pid=29037 refcnt=2 ... etc It looks like spid is what I want. But of course the output from setkey is truncated, so I can't say for definite whether policy 811033 exists or not. It's not in the part of the output that I can see. If I re-kick the tunnels, I can get up to 215 concurrently. Further attempts give more "no policy found" errors. Any more suggestions? Regards, Brian. |
From: KOVACS K. <hi...@ba...> - 2006-03-29 13:56:30
|
Hi, On Wednesday 29 March 2006 14.47, jamal wrote: > On Wed, 2006-29-03 at 13:12 +0100, Brian Candler wrote: > > Thank you - that at least lets me see the 250 sets of policies > > installed. > > > > However there still seem to be problems with racoon communicating with > > the kernel. By retrying some tunnels I was able to get up to 178, but > > after this I start getting these errors: > > > > Mar 29 12:43:13 candlerb racoon: ERROR: no policy found: id:811033. > > Mar 29 12:43:13 candlerb racoon: ERROR: no policy found: id:811057. > > > > (that's when trying to start tunnels 179 and 180) > > The way i read that is it is caused by acquires coming from the kernel > with those two policy ids and racoon being unable to find the policies. > It is as if the kernel has these policies but not racoon which would be > strange if racoon installed them to begin with. IMHO this is probably a byproduct of the pfkey problems. Racoon adds the policy to its database only if it receives a pfkey SPDUPDATE, SPDADD or SPDDUMP message. So if some pfkey messages get lost then racoon's database won't be updated, and so the SPD database of the kernel and that of racoon will be different. -- Regards, Krisztian Kovacs |
From: jamal <ha...@cy...> - 2006-03-29 15:24:34
|
On Wed, 2006-29-03 at 14:51 +0100, Brian Candler wrote: > On Wed, Mar 29, 2006 at 07:47:56AM -0500, jamal wrote: > > Look at src/libipsec/pfkey.c::pfkey_open() > > its set to about 128K at the moment. > > I have now changed that to 1024K. Thanks. > If you tests still indicate that is not high enough, you should probably up it a little more. > Right, I've tested this again. Trying to bring up 250 tunnels, 90 succeeded > at first attempt. I see one error "failed to recv from pfkey (Resource > temporarily unavailable)", as well as 34 instances like > > Mar 29 14:29:08 candlerb racoon: ERROR: no policy found: id:828457 > Mar 29 14:29:08 candlerb racoon: ERROR: no policy found: id:828481 > Krisztian has made a good description why this would be related to the same problem. To quote: On Wed, 2006-29-03 at 15:55 +0200, KOVACS Krisztian wrote: > IMHO this is probably a byproduct of the pfkey problems. Racoon adds >the policy to its database only if it receives a pfkey SPDUPDATE, >SPDADD or SPDDUMP message. So if some pfkey messages get lost then >racoon's database won't be updated, and so the SPD database of the > kernel and that of racoon will be different. Yes, that makes a lot of sense. So change both the receive and transmit buffer sizes. > However I don't know how to get policy ids from ip xfrm; the output looks > like this > > # ip xfrm policy list use -s for more verbosity. ip -s xfrm policy list But i dont think thats needed anymore; your problem is the same one as before. It wont harm to just validate though that we have those policies in the kernel. cheers, jamal |
From: Brian C. <B.C...@po...> - 2006-03-30 15:25:15
|
On Wed, Mar 29, 2006 at 10:23:48AM -0500, jamal wrote: > > IMHO this is probably a byproduct of the pfkey problems. Racoon adds > >the policy to its database only if it receives a pfkey SPDUPDATE, > >SPDADD or SPDDUMP message. So if some pfkey messages get lost then > >racoon's database won't be updated, and so the SPD database of the > > kernel and that of racoon will be different. > > Yes, that makes a lot of sense. > So change both the receive and transmit buffer sizes. They both appear to be set to the same size: pfkey_open() { int so; const int bufsiz = 1024 * 1024; /*is 128K enough?*/ if ((so = socket(PF_KEY, SOCK_RAW, PF_KEY_V2)) < 0) { __ipsec_set_strerror(strerror(errno)); return -1; } /* * This is a temporary workaround for KAME PR 154. * Don't really care even if it fails. */ (void)setsockopt(so, SOL_SOCKET, SO_SNDBUF, &bufsiz, sizeof(bufsiz)); (void)setsockopt(so, SOL_SOCKET, SO_RCVBUF, &bufsiz, sizeof(bufsiz)); __ipsec_errcode = EIPSEC_NO_ERROR; return so; } However this code doesn't check for return values, so as it stands I have no indication of whether it has successfully changed anything. Anyway, I've now changed it to 2048 * 1024. Surely 2MB is enough :-) > use -s for more verbosity. ip -s xfrm policy list > > But i dont think thats needed anymore; your problem is the same one as > before. It wont harm to just validate though that we have those policies > in the kernel. Picking three examples: Mar 30 16:14:59 localhost racoon: ERROR: no policy found: id:5961. Mar 30 16:14:59 localhost racoon: ERROR: no policy found: id:5985. Mar 30 16:14:59 localhost racoon: ERROR: no policy found: id:6009. [root@localhost ~]# ip -s xfrm policy list | grep 5961 dir out action allow index 5961 priority 2147483648 share any flags 0x00000000 [root@localhost ~]# ip -s xfrm policy list | grep 5985 dir out action allow index 5985 priority 2147483648 share any flags 0x00000000 [root@localhost ~]# ip -s xfrm policy list | grep 6009 dir out action allow index 6009 priority 2147483648 share any flags 0x00000000 So yes, the policies are in the kernel, but they are not known to racoon. It may be coincidence, but now I've reset the buffer to 2MB and tested again, I don't see any "failed to recv from pfkey" messages. However, only 34 L2TP/IPSEC tunnels came up initially. After rekicking, I get 215, which I think is the same as the best I got before. I still see more 'no policy found' messages. Anything else I should look at? I tried running 'racoon -d [-d]' but it's far too verbose when you have a large number of policies. For every one policy read it dumps out the entire SPD seen so far, so the amount of syslog output increases as the square of the number of entries in the SPD. Regards, Brian. |
From: KOVACS K. <hi...@ba...> - 2006-03-31 11:03:02
|
Hi, On Thursday 30 March 2006 17.23, Brian Candler wrote: > (void)setsockopt(so, SOL_SOCKET, SO_SNDBUF, &bufsiz, > sizeof(bufsiz)); (void)setsockopt(so, SOL_SOCKET, SO_RCVBUF, &bufsiz, > sizeof(bufsiz)); > > __ipsec_errcode = EIPSEC_NO_ERROR; > return so; > } > > However this code doesn't check for return values, so as it stands I have > no indication of whether it has successfully changed anything. > > Anyway, I've now changed it to 2048 * 1024. Surely 2MB is enough :-) These buffer sizes are also limited by the kernel. Take a look at these sysctls: /proc/sys/net/core/{r,w}mem_{default,max}. No pfkey socket buffers can exceed the maximum value set in {r,w}mem_max, so simply setting those values in racoon to 2MB is probably not enough. Unfortunately you cannot set pfkey buffer size limits separately in Linux, although if you feel this is a problem I have a patch which implements /proc/sys/net/pfkey/{r,w}mem_{default,max}. > I tried running 'racoon -d [-d]' but it's far too verbose when you have a > large number of policies. For every one policy read it dumps out the > entire SPD seen so far, so the amount of syslog output increases as the > square of the number of entries in the SPD. Racoon logging is not very fine grained unfortunately. Profiling racoon with a large number of SPD/SADB entries shows that a lot of time is spent printing these (next to useless) log messages you mention. I've simply commented them out in racoon. By doing this you can achieve a measurable speedup... If you initiate a large number of connections at the same time it's very easy to overload racoon. For example, the Diffie-Hellmann key exchange is computationally -- Regards, Krisztian Kovacs |
From: Patrick M. <ka...@tr...> - 2006-03-31 11:43:26
|
KOVACS Krisztian wrote: > These buffer sizes are also limited by the kernel. Take a look at these > sysctls: /proc/sys/net/core/{r,w}mem_{default,max}. No pfkey socket buffers > can exceed the maximum value set in {r,w}mem_max, so simply setting those > values in racoon to 2MB is probably not enough. > > Unfortunately you cannot set pfkey buffer size limits separately in Linux, > although if you feel this is a problem I have a patch which > implements /proc/sys/net/pfkey/{r,w}mem_{default,max}. There already is a SO_{SND,RCV}BUFFORCE socket option to allow root to override the maximum. For pfkey it doesn't make a difference, for protocols usable by normal users you may not want to increase the system-wide limit. |
From: Brian C. <B.C...@po...> - 2006-03-31 13:02:16
|
On Fri, Mar 31, 2006 at 01:01:58PM +0200, KOVACS Krisztian wrote: > These buffer sizes are also limited by the kernel. Take a look at these > sysctls: /proc/sys/net/core/{r,w}mem_{default,max}. No pfkey socket buffers > can exceed the maximum value set in {r,w}mem_max, so simply setting those > values in racoon to 2MB is probably not enough. # head /proc/sys/net/core/{r,w}mem_{default,max} ==> /proc/sys/net/core/rmem_default <== 110592 ==> /proc/sys/net/core/rmem_max <== 110592 ==> /proc/sys/net/core/wmem_default <== 110592 ==> /proc/sys/net/core/wmem_max <== 110592 > Unfortunately you cannot set pfkey buffer size limits separately in Linux, > although if you feel this is a problem I have a patch which > implements /proc/sys/net/pfkey/{r,w}mem_{default,max}. Is it OK if I just do # echo 1048576 >/proc/sys/net/core/rmem_max # echo 1048576 >/proc/sys/net/core/wmem_max ? If not, please point me at your patch. Cheers, Brian. |
From: Brian C. <B.C...@po...> - 2006-03-31 14:46:48
|
On Fri, Mar 31, 2006 at 02:01:44PM +0100, Brian Candler wrote: > Is it OK if I just do > > # echo 1048576 >/proc/sys/net/core/rmem_max > # echo 1048576 >/proc/sys/net/core/wmem_max Hey, this works :-) I got up to 250 tunnels straight away. However, things get tricky around 420 tunnels: lots of messages like INFO: IPsec-SA expired: ESP/Transport X.X.X.X[0]->Y.Y.Y.Y[0] spi=... WARNING: the expire message is received but the handler has not been established ERROR: X.X.X.X give up to get IPsec-SA due to time up to wait. ERROR: X.X.X.X give up to get IPsec-SA due to time up to wait. However, by tickling gently I am currently up to 446. At this point, it looks like the Cisco has run out of RAM (it has 128MB). So I won't be able to probe any further until my test 7204 has been upgraded... FYI: based on what I've seen so far, on average each additional L2TP/IPSEC tunnel uses about 32KB of RAM on the Cisco. This is the figure I was most keen to determine. Thanks to everyone who has helped me get to this position; I appreciate your help very much. Regards, Brian. |