(This is in reference to the Sidewinder v7 VPN interoperability thread on the mailing list.)
I have racoon set up on a Mac OS X machine and during phase 2 it emits a single proposal with eight transforms numbered 1 through 8. The Sidewinder it is connecting to selects a single transform, returns a proposal numbered 1 but the transform is numbered 0, so racoon decides the proposal comparison fails (despite the Sidewinder selecting one of the proposals the Mac presents it with.)
From RFC 2048, section 4.2:
"When responding to a Security Association payload, the responder MUST send a Security Association payload with the selected proposal, which may consist of multiple Proposal payloads and their associated Transform payloads. Each of the Proposal payloads MUST contain a single Transform payload associated with the Protocol. The responder SHOULD retain the Proposal # field in the Proposal payload and the Transform # field in each Transform payload of the selected Proposal. Retention of Proposal and Transform numbers should speed the initiator's protocol processing by negating the need to compare the respondor's selection with every offered option. These values enable the initiator to perform the comparison directly and quickly."
Note "SHOULD" - this means that the transform comparison, done by index matching in ipsec_doi.c, is incorrect; the transform comparison should be done based on the enclosed transform properties against the selected transforms that were sent out instead of the index number returned by the remote end.
Alternatively, perhaps the transform comparison by property values should be performed as a last resort in the case the transform indexes fail to compare properly?