Theses two constants are defined to the same value under src/racoon/oakley.h:
#define OAKLEY_ATTR_AUTH_METHOD_XAUTH_PSKEY_I 65001
#define OAKLEY_ATTR_AUTH_METHOD_GSSAPI_KRB 65001
When racoon is compiled with both --enable-hybrid and --enable-gssapi GSSAPI authentication breaks since racoon decides its actually referring to the XAUTH_PSKEY authentication mode.
By changing the OAKLEY_ATTR_AUTH_METHOD_XAUTH constants to differ from the OAKLEY_ATTR_AUTH_METHOD_GSSAPI constants I was able to compile with both --enable-hybrid and --enable-gssapi with working GSSAPI authentication.
Log entries of the problem when authenticating with GSSAPI:
2006-02-09 15:40:44: DEBUG: begin QUICK mode.
2006-02-09 15:40:44: ERROR: Hybrid auth negotiated but peer did not announced as Xauth capable
2006-02-09 15:40:44: ERROR: Attempt to start phase 2 whereas Xauth failed
2006-02-09 15:40:44: ERROR: failed to begin ipsec sa negotication.