ocf-linux-users

[Ocf-linux-users] OCF/OpenSwan tunnel crashes

[Ronan M. <ron...@gm...>]

Hello,

I seem to be having some problems with setting OpenSwan (2.4.9) up on
OCF(20070727) (using the lastest openSwan patch) and then up setting a VPN
tunnel up.

I have a RHEL5 system using a 2.6.18 kernel.

I can setup a host to host connection and the tunnel works fine with pings,
scp and ftp all working through the tunnel for both cryptosoft and my
hardware accelerator.

My problems arise when I try to setup  the two hosts as gateways. When I
setup a system where packets are coming from somewhere else on the network
and are allowed to pass through the VPN (via the connection details in
ipsec.conf) I get system crashes for cryptosoft and my hardware driver ( I
can get no ouput log files :( ).

>From what I have read on other forums some kind of race condition in OCF
could be causing this. Has anyone else seen these kind of issues. There was
a recommendation to try setting the USE_CBIMM flag to zero in ipsec_ocf.c
which would disable immediate callbacks. This may resolve an appartently
know race condition between the network stack and OCF! Is this true. What
impact on performance would this action have

-- 
Regards,
Ronan

Re: [Ocf-linux-users] OCF/OpenSwan tunnel crashes

[Ronan M. <ron...@gm...>]

I have replicated this crash with cryptosoft. The systems falls over during
a 10 second blast of 6.8 Mbits from Smartbits. This usiing using OpenSwan (
2.4.9) up on OCF(20070727) (using the lastest openSwan patch) and then up
setting a VPN tunnel.

The system details are
Kernel 2.6.18.8-EL5

model name: Intel(R) Pentium(R) 4 CPU 2.40GHz

cpu MHz: 2392.119, cache size: 512 KB, stepping: 7
The crash log is below

BUG: soft lockup detected on CPU#0!

 [<c044a0b7>] softlockup_tick+0x98/0xa6

 [<c042cc98>] update_process_times+0x39/0x5c

 [<c04176ec>] smp_apic_timer_interrupt+0x5c/0x64

 [<c04049bf>] apic_timer_interrupt+0x1f/0x24

 [<c05fc458>] _spin_lock+0x7/0xf

 [<e1021e13>] ipsec_tunnel_SAlookup+0x40/0x561 [ipsec]

 [<e102243c>] ipsec_tunnel_start_xmit+0x108/0x154 [ipsec]

 [<c05a3d8a>] dev_hard_start_xmit+0x198/0x1ee

 [<c05b1a7a>] __qdisc_run+0xe4/0x19a

 [<c05a557b>] dev_queue_xmit+0x144/0x24d

 [<c05a7b50>] neigh_connected_output+0x79/0x8c

 [<c05c2755>] ip_output+0x1ce/0x1f9

 [<c05bef8b>] ip_forward+0x1d9/0x22e

 [<c05bddc5>] ip_rcv+0x3ef/0x429

 [<c05a39b9>] netif_receive_skb+0x2c1/0x339

 [<e08d4c41>] e1000_clean_rx_irq+0x3ef/0x49b [e1000]

 [<e08d33c5>] e1000_clean+0x69/0x106 [e1000]

 [<c05a5354>] net_rx_action+0x92/0x175

 [<c04281cf>] __do_softirq+0x5a/0xbb

 [<c0406461>] do_softirq+0x52/0x9d

 [<c0406406>] do_IRQ+0xa5/0xae

 [<c040492e>] common_interrupt+0x1a/0x20

 [<c04df347>] memcpy+0x17/0x28

 [<e1021d3d>] ipsec_tunnel_xsm_complete+0x7c/0x112 [ipsec]

 [<e1022c66>] ipsec_xsm+0x123/0x126 [ipsec]

 [<e1031e2e>] ipsec_ocf_xmit_cb+0x10c/0x111 [ipsec]

 [<e0b03e76>] crypto_done+0x72/0x110 [ocf]

 [<e0b29cf9>] swcr_process+0xa04/0xa2e [cryptosoft]

 [<e0b03fe2>] crypto_invoke+0xce/0xd4 [ocf]

 [<c0434ef8>] finish_wait+0x2a/0x4b

 [<e0b0477b>] crypto_proc+0x15d/0x4c7 [ocf]

 [<c0434e65>] autoremove_wake_function+0x0/0x2d

 [<e0b0461e>] crypto_proc+0x0/0x4c7 [ocf]

 [<c0404c3b>] kernel_thread_helper+0x7/0x10

Has anyone else seen this crash?

On 07/09/2007, Ronan Morrissey <ron...@gm...> wrote:
>
> Hello,
>
> I seem to be having some problems with setting OpenSwan (2.4.9) up on
> OCF(20070727) (using the lastest openSwan patch) and then up setting a VPN
> tunnel up.
>
> I have a RHEL5 system using a 2.6.18 kernel.
>
> I can setup a host to host connection and the tunnel works fine with
> pings, scp and ftp all working through the tunnel for both cryptosoft and my
> hardware accelerator.
>
> My problems arise when I try to setup  the two hosts as gateways. When I
> setup a system where packets are coming from somewhere else on the network
> and are allowed to pass through the VPN (via the connection details in
> ipsec.conf) I get system crashes for cryptosoft and my hardware driver ( I
> can get no ouput log files :( ).
>
> From what I have read on other forums some kind of race condition in OCF
> could be causing this. Has anyone else seen these kind of issues. There was
> a recommendation to try setting the USE_CBIMM flag to zero in ipsec_ocf.c
> which would disable immediate callbacks. This may resolve an appartently
> know race condition between the network stack and OCF! Is this true. What
> impact on performance would this action have
>
> --
> Regards,
> Ronan




-- 
Regards,
Ronan

Re: [Ocf-linux-users] OCF/OpenSwan tunnel crashes

[David M. <Dav...@se...>]

Jivin Ronan Morrissey lays it down ...
> I have replicated this crash with cryptosoft. The systems falls over during
> a 10 second blast of 6.8 Mbits from Smartbits. This usiing using OpenSwan (
> 2.4.9) up on OCF(20070727) (using the lastest openSwan patch) and then up
> setting a VPN tunnel.

The call sequence looks pretty obvious.

Definately try setting USE_CBIMM to 0.

It may have some impact on performance,  but for loaded systems
it may actually be better.  Besides,  hangs are bad for performance as
well.

cryptosoft could be more of a trigger here as well since it is actually
SW it can callback before returning from crypto_process,  which is still
possible with HW,  but much less likely.

Do you have kernel preemption enabled ?  Is it possible to try turning
it off if it is ?

Cheers,
Davidm

> The system details are
> Kernel 2.6.18.8-EL5
> 
> model name: Intel(R) Pentium(R) 4 CPU 2.40GHz
> 
> cpu MHz: 2392.119, cache size: 512 KB, stepping: 7
> The crash log is below
> 
> BUG: soft lockup detected on CPU#0!
> 
>  [<c044a0b7>] softlockup_tick+0x98/0xa6
>  [<c042cc98>] update_process_times+0x39/0x5c
>  [<c04176ec>] smp_apic_timer_interrupt+0x5c/0x64
>  [<c04049bf>] apic_timer_interrupt+0x1f/0x24
>  [<c05fc458>] _spin_lock+0x7/0xf
>  [<e1021e13>] ipsec_tunnel_SAlookup+0x40/0x561 [ipsec]
>  [<e102243c>] ipsec_tunnel_start_xmit+0x108/0x154 [ipsec]
>  [<c05a3d8a>] dev_hard_start_xmit+0x198/0x1ee
>  [<c05b1a7a>] __qdisc_run+0xe4/0x19a
>  [<c05a557b>] dev_queue_xmit+0x144/0x24d
>  [<c05a7b50>] neigh_connected_output+0x79/0x8c
>  [<c05c2755>] ip_output+0x1ce/0x1f9
>  [<c05bef8b>] ip_forward+0x1d9/0x22e
>  [<c05bddc5>] ip_rcv+0x3ef/0x429
>  [<c05a39b9>] netif_receive_skb+0x2c1/0x339
>  [<e08d4c41>] e1000_clean_rx_irq+0x3ef/0x49b [e1000]
>  [<e08d33c5>] e1000_clean+0x69/0x106 [e1000]
>  [<c05a5354>] net_rx_action+0x92/0x175
>  [<c04281cf>] __do_softirq+0x5a/0xbb
>  [<c0406461>] do_softirq+0x52/0x9d
>  [<c0406406>] do_IRQ+0xa5/0xae
>  [<c040492e>] common_interrupt+0x1a/0x20
>  [<c04df347>] memcpy+0x17/0x28
>  [<e1021d3d>] ipsec_tunnel_xsm_complete+0x7c/0x112 [ipsec]
>  [<e1022c66>] ipsec_xsm+0x123/0x126 [ipsec]
>  [<e1031e2e>] ipsec_ocf_xmit_cb+0x10c/0x111 [ipsec]
>  [<e0b03e76>] crypto_done+0x72/0x110 [ocf]
>  [<e0b29cf9>] swcr_process+0xa04/0xa2e [cryptosoft]
>  [<e0b03fe2>] crypto_invoke+0xce/0xd4 [ocf]
>  [<c0434ef8>] finish_wait+0x2a/0x4b
>  [<e0b0477b>] crypto_proc+0x15d/0x4c7 [ocf]
>  [<c0434e65>] autoremove_wake_function+0x0/0x2d
>  [<e0b0461e>] crypto_proc+0x0/0x4c7 [ocf]
>  [<c0404c3b>] kernel_thread_helper+0x7/0x10
> 
> Has anyone else seen this crash?
> 
> On 07/09/2007, Ronan Morrissey <ron...@gm...> wrote:
> >
> > Hello,
> >
> > I seem to be having some problems with setting OpenSwan (2.4.9) up on
> > OCF(20070727) (using the lastest openSwan patch) and then up setting a VPN
> > tunnel up.
> >
> > I have a RHEL5 system using a 2.6.18 kernel.
> >
> > I can setup a host to host connection and the tunnel works fine with
> > pings, scp and ftp all working through the tunnel for both cryptosoft and my
> > hardware accelerator.
> >
> > My problems arise when I try to setup  the two hosts as gateways. When I
> > setup a system where packets are coming from somewhere else on the network
> > and are allowed to pass through the VPN (via the connection details in
> > ipsec.conf) I get system crashes for cryptosoft and my hardware driver ( I
> > can get no ouput log files :( ).
> >
> > From what I have read on other forums some kind of race condition in OCF
> > could be causing this. Has anyone else seen these kind of issues. There was
> > a recommendation to try setting the USE_CBIMM flag to zero in ipsec_ocf.c
> > which would disable immediate callbacks. This may resolve an appartently
> > know race condition between the network stack and OCF! Is this true. What
> > impact on performance would this action have
> >
> > --
> > Regards,
> > Ronan
> 
> 
> 
> 
> -- 
> Regards,
> Ronan

> -------------------------------------------------------------------------
> This SF.net email is sponsored by: Microsoft
> Defy all challenges. Microsoft(R) Visual Studio 2005.
> http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
> _______________________________________________
> Ocf-linux-users mailing list
> Ocf...@li...
> https://lists.sourceforge.net/lists/listinfo/ocf-linux-users


-- 
David McCullough,  dav...@se...,   Ph:+61 734352815
Secure Computing - SnapGear  http://www.uCdot.org http://www.cyberguard.com