Hello ipsec-tools mailing list,

I believe I've found a bug with racoon that is trigged when compiled with the --enable-hybrid and --enable-gssapi flags. 

The bug itself is below (a patch to fix is available at the end of this message):

if ((iph1->mode_cfg->flags & ISAKMP_CFG_VENDORID_XAUTH) == 0) {
           "Hybrid auth negotiated but peer did not "
           "announced as Xauth capable\n");
       return -1;

Both the xauth extension [draft-ietf-ipsec-isakmp-xauth-06] and GSSAPI extension [http://tools.ietf.org/id/draft-ietf-ipsec-isakmp-gss-auth-07.txt] to ipsec use the private authentication identifier range starting at 65001.

If the xauth code sees one of its authentication ids being used with a different vendor code it returns -1 and the phase2 negotiation fails.  This is incorrect behavior as other authentication extensions differentiated by vendor ids share the same private authentication.  The xauth code is currently aborting the phase2 negotiation before any other authentication extension can run.

I believe the correct behavior should be to warn the user that something fishy might be going on and return 0 from this function. This is the same behavior that a non-private authentication code would have.  Once returned the phase2 negotiation could continue with other authentication extensions (GSSAPI). 

I originally encountered this problem on a Debian Etch system attempting to do GSSAPI authentication with ipsec-tools 0.6.6-3.1etch1.  The problem still appears to be in the latest ipsec-tools release.  Below is a patch that I have tested against 0.6.6-3.1etch1 with success:

--- isakmp_xauth.c      2006-06-12 14:06:01.000000000 -0400
+++ /afs/metacarta.com/user/gharris/isakmp_xauth.c      2008-03-28 17:46:47.000000000 -0400
@@ -742,10 +742,14 @@
                if ((iph1->mode_cfg->flags & ISAKMP_CFG_VENDORID_XAUTH) == 0) {
-                       plog(LLV_ERROR, LOCATION, NULL,
+                       plog(LLV_WARNING, LOCATION, NULL,
                            "Hybrid auth negotiated but peer did not "
-                           "announced as Xauth capable\n");
-                       return -1;
+                           "announced as Xauth capable -- possibly a"
+                            "different authentication extension\n");
+                        /* we must return 0 to allow other extensions
+                         * overloading the private authentication ids
+                         * a chance to run */
+                       return 0;

                if (xst->status != XAUTHST_OK) {

Thanks for the otherwise excellent tools!

Greg Harris
System Administrator
MetaCarta, Inc.