I am running racoon on a Linux 2.6.22 host. I am trying to use IPsec to a Solaris 5.9 using it's in.iked daemon.
When I try to ping the Solaris, it fails with the error: "Phase 1 negotiation error: invalid payload type".
The trace shows an ISAKMP Informational packet from the Solaris to the effect of:

Internet Security Association and Key Management:
Initiator cookie: (num)
Responder cookie: (num)
Next payload: Notification (11)
Version 1.0
Exchange type: Informational (5)

Notification Payload:
Next payload: None (0)
Domain of Interpretation: IPSEC (1)
Protocol ID: ISAKMP (1)
SPI size: 16

Obviously, racoon is passing a payload type that the Solaris does not support.
Is there a setting in the racoon.conf that I need to set (or unset) that will help me through this?

Phil Bellino


Phil Bellino
MRV Communications, Inc.
Boston Product Division
295 Foster St.
Littleton,MA 01460
Tel: (978)952-4807
Email: pbellino@mrv.com