Hello,
 
I am trying to create a VPN tunnel between a RedHat ES 4 Linux with ipsec-tools 0.5 and between a appliance card running
vxWorks with IKE/IPSec or between the Linux and a Check Point R55 VPN-1 getway.

I started working on creating a tunnel between the Linux and the hardware appliance running vxWorks+IPSec, phase 1 (main
mode) completed fine, but phase 2 did not.

I used 3DES/MD5 for main mode and DES/MD5 for quick mode.

In the console of the appliance I saw the debug messages of the vxWorks IKE :

IKE: Negotiating for:
        Local ID: NO ID types exchanged <192.168.1.135>
        Peer ID: NO ID types exchanged <192.168.1.144>
        Protocol: 0, PeerPort: 0, LocalPort: 0

Which shows that quick mode packet 1 has no ID fields in it (the field that determine for which hosts or subnets
this exchnage is taking place).

I used an Check Point R55 VPN-1 gateway on which I can run better vpn debug (vpn debug trunc), and saw using the
ike.elg debug file a few interesting facts :

First test - the Linux initiates the traffic, so it will initiate IKE

1. The security association contains two proposals in quick mode packet 1 -

   - PROTO_IPSEC_AH
   - PROTO_IPSEC_ESP

I do not know why does the Linux send an AH proposal when I want only ESP - can this be configured somewhere ?

2. In the ESP proposal I see one transform payload :


Here is the ike.elg data :


Transform Payload - ESP_DES

Next Payload: NONE
Reserved: 0
Length:  00 1c  (28)
TransNum: 1
TransId: 2
Reserved2: 00 00  (0)

SA Life Type:  Seconds
SA Life Duration: 3600
Encapsulation Mode: Transport
Authentication Alg: HMAC-MD5
Group Description: Alternate 1024-bit MODP group

There are no additional payloads after this one.

right after this quick mode packet 1 I can see that the linux replies with a notification :

Notify Type: 14 (NO-PROPOSAL-CHOSEN)


In the above test, the linux was the initiator of the traffic.

If the VPN-1 is the initiator, I can see the following in quick mode (main mode ends fine) :

The security association contains one proposal which contains one transform payload :


Here again the data from ike.elg


Transform Payload - ESP_DES

Next Payload: NONE
Reserved: 0
Length:  00 20  (32)
TransNum: 1
TransId: 2
Reserved2: 00 00  (0)

Group Description: Alternate 1024-bit MODP group
SA Life Type:  Seconds
SA Life Duration: 3600
Authentication Alg: HMAC-MD5
Encapsulation Mode: Tunnel


I see the following payloads after this one :

- nonce
- key
- ID  - identification payload contains the ip address of the VPN-1 gateway
- ID  - identification payload contains the ip address of the Linux server

These 4 payloads do not exist in quick mode packet 1 which comes from the Linux when it is
the initiator of the quick mode.

After this quick mode packet 1 (which comes from the VPN-1), I cann see that the Linux sends a notification message :

Notify Type: 14 (NO-PROPOSAL-CHOSEN)


I also tried to add the following to the 'remote X.X.X.X {...}' definitions :

peers_identifier address;
verify_identifier on;
support_proxy on;

But nothing helps, I still get the messages (I see these when I run racoon in debug mode) :


2006-07-18 17:56:31: ERROR: unknown notify message, no phase2 handle found.
2006-07-18 17:56:31: DEBUG: notification message 14:NO-PROPOSAL-CHOSEN, doi=1 proto_id=3 spi=00000000(size=4).


And these messages in /var/log/messages :

Jul 18 17:56:31 blue racoon: 2006-07-18 17:56:31: INFO: initiate new phase 2 negotiation: 192.168.1.144[0]<=>192.168.1.210[0]
Jul 18 17:56:31 blue racoon: 2006-07-18 17:56:31: ERROR: unknown notify message, no phase2 handle found.