Ok, I think I've finally gotten to the bottom of this. When the first
tunnel gets established, I have two SAD entries, like so:
a.b.c.d x.y.z.w, spi=1
x.y.z.w a.b.c.d, spi=2
this is for communicating from my server a.b.c.d/32 to the remote network
When the next tunnel gets established, I have two more:
a.b.c.d x.y.z.w, spi=3
x.y.z.w a.b.c.d, spi=4
these are for communicating from my server a.b.c.d/32 to the remote network
Now the problem is the dumb vpn (fortinet router) on the other side uses the
new spi, 4, regardless of the host I'm connecting to. So if I connect to
10.2.2.7, it uses spi 3 for outgoing and spi 4 for incoming, which is
correct. But if I connect to 192.168.0.7, it uses spi 1 for outgoing and
spi 4 for incoming!
So the Fortinet router blindly uses the latest SA that has been established,
spi #4, even though it is for a different tunnel! It never goes back to
using spi #2.
But my LAN computers (192.168.8.0/24) continue to work, probably because
MASQUERADE filter is so smart that it dynamically decides that SPI 4 goes
with SPI 1.
So my question is, how can I fix this? I can't tell my workplace to change
what they are doing. If there was someway to just use a single association
for both tunnels, that would do the trick. Or if there were someway to tell
the kernel not to drop my packet because even though the spi isn't the
matching one, that would probably do it too.