Hi,

I've set up two machines, installed ipsec-tools & racoon on them.
The network topology is the following:

SERVER <---->{INTERNET}----[NAT]<--> CLIENT

Server has 1:0.7.1-1.6ubuntu1 version, client has 1:0.8.0-9ubuntu1.

The 4 differences in configs: admin has anonymous remote, client has named, and server config also contains:   generate_policy on;  passive on;.
The client has the spds autogenerated by its IP and manually set:

flush;
spdflush;
spdadd 10.59.145.240  212.52.SS.SS any -P out ipsec esp/transport//require;
spdadd 212.52.SS.SS 10.59.145.240  any -P in ipsec esp/transport//require;

On the server, the policies are autogenerated:

# setkey -D
212.52.SS.SS[4500] 79.125.CC.CC[4500] 
        esp-udp mode=transport spi=10127209(0x009a8769) reqid=0(0x00000000)
        E: 3des-cbc  c216371d 4363aa86 99ad7dce 21508bad 96fd124d 90b32e07
        A: hmac-sha1  f61d8e45 5dfd55f8 5628db12 7896f107 f782fece
        seq=0x00000000 replay=4 flags=0x00000000 state=mature 
        created: Jun 22 13:56:50 2012   current: Jun 22 13:57:03 2012
        diff: 13(s)     hard: 120(s)    soft: 96(s)
        last:                           hard: 0(s)      soft: 0(s)
        current: 0(bytes)       hard: 0(bytes)  soft: 0(bytes)
        allocated: 0    hard: 0 soft: 0
        sadb_seq=1 pid=30633 refcnt=0
79.125.CC.CC[4500] 212.52.SS.SS[4500] 
        esp-udp mode=transport spi=46172739(0x02c08a43) reqid=0(0x00000000)
        E: 3des-cbc  e5b705c0 777f2bc8 27355513 f26030e5 71bf0ed6 44ab2a84
        A: hmac-sha1  85f6af62 910b6004 3a76b7c2 fbd45c77 ca4e7d41
        seq=0x00000000 replay=4 flags=0x00000000 state=mature 
        created: Jun 22 13:56:50 2012   current: Jun 22 13:57:03 2012
        diff: 13(s)     hard: 120(s)    soft: 96(s)
        last: Jun 22 13:56:51 2012      hard: 0(s)      soft: 0(s)
        current: 832(bytes)     hard: 0(bytes)  soft: 0(bytes)
        allocated: 13   hard: 0 soft: 0
        sadb_seq=0 pid=30633 refcnt=0

In this setup, the server gets the encrypted packets, decodes them, but sends the replies unencrypted.
snippet from ping & tcpdump:

14:07:03.996976 IP 79.125.CC.CC.4500 > 212.52.SS.SS.4500: UDP-encap: ESP(spi=0x0c750c76,seq=0x1), length 100
14:07:03.997056 IP 212.52.SS.SS > 79.125.CC.CC: ICMP echo reply, id 6728, seq 1192, length 64

This is the main problem.
If I flush the generated policies, and add them manually everything works fine:

setkey -c
flush;
spdflush;
spdadd 212.52.SS.SS 79.125.CC.CC any -P out ipsec esp/transport//require;
spdadd 79.125.CC.CC 212.52.SS.SS any -P in ipsec esp/transport//require;

# setkey -D
212.52.SS.SS[4500] 79.125.CC.CC[4500] 
        esp-udp mode=transport spi=107727401(0x066bca29) reqid=0(0x00000000)
        E: 3des-cbc  724cdf41 6c174375 215c202f f0d5281a 59e5c41a 8903a983
        A: hmac-sha1  cbb2ea42 46769cfb 8829e40e e4ea2f66 c707a566
        seq=0x00000000 replay=4 flags=0x00000000 state=mature 
        created: Jun 22 13:59:28 2012   current: Jun 22 13:59:34 2012
        diff: 6(s)      hard: 120(s)    soft: 96(s)
        last: Jun 22 13:59:29 2012      hard: 0(s)      soft: 0(s)
        current: 320(bytes)     hard: 0(bytes)  soft: 0(bytes)
        allocated: 5    hard: 0 soft: 0
        sadb_seq=1 pid=30680 refcnt=0
79.125.CC.CC[4500] 212.52.SS.SS[4500] 
        esp-udp mode=transport spi=212390670(0x0ca8d30e) reqid=0(0x00000000)
        E: 3des-cbc  2c54c6cb 9eb05480 f7115aa2 e5de5543 70eea0c3 53ab8c35
        A: hmac-sha1  8821f865 b86f9bc2 bfa0bc73 f13a19e9 5f84af4b
        seq=0x00000000 replay=4 flags=0x00000000 state=mature 
        created: Jun 22 13:59:28 2012   current: Jun 22 13:59:34 2012
        diff: 6(s)      hard: 120(s)    soft: 96(s)
        last: Jun 22 13:59:29 2012      hard: 0(s)      soft: 0(s)
        current: 320(bytes)     hard: 0(bytes)  soft: 0(bytes)
        allocated: 5    hard: 0 soft: 0
        sadb_seq=0 pid=30680 refcnt=0

tcpdump & ping:
14:06:59.995218 IP 79.125.CC.CC.4500 > 212.52.SS.SS.4500: UDP-encap: ESP(spi=0x0f5c08df,seq=0x44), length 100
14:06:59.995328 IP 212.52.SS.SS.4500 > 79.125.CC.CC.4500: UDP-encap: ESP(spi=0x052fa624,seq=0x44), length 100

What am I missing? Is it some configuration issue, or is it a bug in racoon policy generation?

Here is config for client:

log notify;
path certificate "/etc/certs";

timer
{
         natt_keepalive 15 seconds;
}

remote 212.52.SS.SS
{
  exchange_mode main;

  my_identifier asn1dn;
  peers_identifier asn1dn;
  verify_identifier on;
  certificate_type x509 "client.crt" "client.key";

  nat_traversal force;
  dpd_delay 30;
  ike_frag on;

  lifetime time 2 min;
  initial_contact on;
  proposal_check obey;

  proposal {
(...)
  }
}

sainfo anonymous
{
(...)
}

And here is for server:

log info;
path certificate "/etc/racoon/certs";

remote anonymous
{
  exchange_mode main;

  my_identifier asn1dn;
  peers_identifier asn1dn;
  verify_identifier on;
  certificate_type x509 "server.cert" "server.key";

  generate_policy on;
  passive on;
  nat_traversal on;
  dpd_delay 30;
  ike_frag on;

  lifetime time 2 min;
  proposal_check claim;

  proposal {
(...)
  }
}

sainfo anonymous
{
(...)
}

Thanks for any helpful comments in advance,
Márton