I am running racoon on a Linux 2.6.22 host. I am trying to use IPsec to a Solaris 5.9 using it's in.iked daemon.
When I try to ping the Solaris, it fails with the error: "Phase 1 negotiation error: invalid payload type".
The trace shows an ISAKMP Informational packet from the Solaris to the effect of:
Internet Security Association and Key Management:
Initiator cookie: (num)
Responder cookie: (num)
Next payload: Notification (11)
Exchange type: Informational (5)
Next payload: None (0)
Domain of Interpretation: IPSEC (1)
Protocol ID: ISAKMP (1)
SPI size: 16
Message Type: INVALID-PAYLOAD-TYPE (1)
Obviously, racoon is passing a payload type that the Solaris does not support.
Is there a setting in the racoon.conf that I need to set (or unset) that will help me through this?
MRV Communications, Inc.
Boston Product Division
295 Foster St.