Hi All,

We are running l2tpns/ipsec-tools VPN server with support for windows_xp/windows_2003 clients and everithing looks fine. (ipsec-tools from CVS and the method of authenticating clients is X.509). Recently I started to test windows_vista VPN client and I noticed that:

  1. the first connection succeeded;

  2. after stopping the existing connection, the second attempt to establish connection failed with the error 789 : .The l2tp connection attempt failed because the security layer encountered a processing error during initial negotiations with the remote computer..

  3. I could only establish the vpv connection again when:

    a) SAs that were created during the first connection is deleted from command line

    b) or life time of SAs on linux side is expaired (~2880 sec ?).


I used oakley log on windows side and racoon debug log on server side to trace error message and I found out the next interesting things:

  1. vista client: uses RFC 3947 whereas windows_xp draft-02;

  2. racoon(during the first connection): after finishing phase 1(main mode) of negotiation with a partner, sends an Informational Notify like this:


2008-04-30 09:06:28: DEBUG: compute IV for phase2

2008-04-30 09:06:28: DEBUG: phase1 last IV:

2008-04-30 09:06:28: DEBUG:

5f2c9ab4 1c192697 e8ce8836

2008-04-30 09:06:28: DEBUG: hash(sha1)

2008-04-30 09:06:28: DEBUG: encryption(3des)

2008-04-30 09:06:28: DEBUG: phase2 IV computed:

2008-04-30 09:06:28: DEBUG:

9a886da4 7d10f0aa

2008-04-30 09:06:28: DEBUG: HASH with:

2008-04-30 09:06:28: DEBUG:

e8ce8836 0000001c 00000001 01106002 6de3089f 7dd97dcc 98f72289 19d6c640

2008-04-30 09:06:28: DEBUG: hmac(hmac_sha1)

2008-04-30 09:06:28: DEBUG: HASH computed:

2008-04-30 09:06:28: DEBUG:

2ade2eed 906177a2 87195fa9 6e241a5d 6fbac674

2008-04-30 09:06:28: DEBUG: begin encryption.

2008-04-30 09:06:28: DEBUG: encryption(3des)

2008-04-30 09:06:28: DEBUG: pad length = 4

2008-04-30 09:06:28: DEBUG:

0b000018 2ade2eed 906177a2 87195fa9 6e241a5d 6fbac674 0000001c 00000001

01106002 6de3089f 7dd97dcc 98f72289 19d6c640 ede1c703

2008-04-30 09:06:28: DEBUG: encryption(3des)

2008-04-30 09:06:28: DEBUG: with key:

2008-04-30 09:06:28: DEBUG:

6342500a ebdaf5b5 345d5738 3977812b 5ce5bc94 b19be869

2008-04-30 09:06:28: DEBUG: encrypted payload by IV:

2008-04-30 09:06:28: DEBUG:

9a886da4 7d10f0aa

2008-04-30 09:06:28: DEBUG: save IV for next:

2008-04-30 09:06:28: DEBUG:

cd97ad91 a475ecfe

2008-04-30 09:06:28: DEBUG: encrypted.

2008-04-30 09:06:28: DEBUG: 84 bytes from 10.0.0.151[500] to 10.0.0.124[500]

2008-04-30 09:06:28: DEBUG: sockname 10.0.0.151[500]

2008-04-30 09:06:28: DEBUG: send packet from 10.0.0.151[500]

2008-04-30 09:06:28: DEBUG: send packet to 10.0.0.124[500]

2008-04-30 09:06:28: DEBUG: src4 10.0.0.151[500]

2008-04-30 09:06:28: DEBUG: dst4 10.0.0.124[500]

2008-04-30 09:06:28: DEBUG: 1 times of 84 bytes message will be sent to 10.0.0.124[500]

2008-04-30 09:06:28: DEBUG:

6de3089f 7dd97dcc 98f72289 19d6c640 08100501 e8ce8836 00000054 d840a817

e31f573d a958babe 9f53faa4 71f5c0e7 07f5ab4b 490d0bf4 b11d66c2 21cc3626

4695c7ba f141493a a6c9c11b cd97ad91 a475ecfe

2008-04-30 09:06:28: DEBUG: sendto Information notify.

2008-04-30 09:06:28: DEBUG: IV freed

2008-04-30 09:06:28: INFO: ISAKMP-SA established 10.0.0.151[500]-10.0.0.124[500] spi:6de3089f7dd97dcc:98f7228919d6c640

2008-04-30 09:06:28: DEBUG: ===


  1. vista client(during the first connection) receives the notify message and continues a quick mode. Here is a procedure of the exchange in both modes for the succeeded connection:

Main mode

Initiator (vista client) Responder (Linux)

------------ ------------

HDR, SA, VID -->

<-- HDR, SA, VID

HDR, KE, Ni, NAT-D, NAT-D -->

<-- HDR, KE, Nr, NAT-D, NAT-D

HDR*#, IDii, [CERT, ] SIG_I -->

<-- HDR*#, IDir, [CERT, ], SIG_R

<-- 7-MM* send MSG


Quick Mode


1-QM* SA, Nonce, Id,Id -->

8-MM receive MSG

<-- 2-QM* SA, Nonce, Id

3-QM* Hash -->


  1. racoon(during the second connection): after finishing phase 1(main mode) of negotiation with a partner, does not send the Informational Notify.

  2. vista client(during the second connection): resends the first message of the quick mode in loop. ( probably, waiting for the 7-MM message).

The most visible solutions for this problem are:

a) always send the notification after finishing phase 1 OR

b) delete the created SAs on server immidiately after disconnecting.

but I am not sure that these are right and compliance with RFC 3947.


Does anybody has an idea how to fix the problem in the correct way?



Thanks,

Valery