Ok, I think I've finally gotten to the bottom of this. When the first tunnel gets established, I have two SAD entries, like so:
a.b.c.d x.y.z.w, spi=1
x.y.z.w a.b.c.d, spi=2
this is for communicating from my server
a.b.c.d/32 to the remote network 192.168.0.0/21.
When the next tunnel gets established, I have two more:
a.b.c.d x.y.z.w, spi=3
x.y.z.w a.b.c.d, spi=4
these are for communicating from my server a.b.c.d/32 to the remote network 10.2.2.0/24
Now the problem is the dumb vpn (fortinet router) on the other side uses the new spi, 4, regardless of the host I'm connecting to. So if I connect to
10.2.2.7, it uses spi 3 for outgoing and spi 4 for incoming, which is correct. But if I connect to 192.168.0.7, it uses spi 1 for outgoing and spi 4 for incoming!
So the Fortinet router blindly uses the latest SA that has been established, spi #4, even though it is for a different tunnel! It never goes back to using spi #2.
But my LAN computers (
192.168.8.0/24) continue to work, probably because MASQUERADE filter is so smart that it dynamically decides that SPI 4 goes with SPI 1.
So my question is, how can I fix this? I can't tell my workplace to change what they are doing. If there was someway to just use a single association for both tunnels, that would do the trick. Or if there were someway to tell the kernel not to drop my packet because even though the spi isn't the matching one, that would probably do it too.