I have had a virtual firewall running in a VM for 2 years without any problems, on ESX 4.1U2. After upgrading the hypervisor to ESXi 5.1 (with latest patches) the same virtual firewall starts currupting random TCP streams.
The guest OS is Gentoo, running plain vanilla kernel 3.0.57 and 3.4.24 (and 2.6.32.x prior).
I have two customers with similar setups, that run without problems. There is only one difference in the setup:
- Working firewalls have one pNIC as uplink for a single vSwitch, the guest connected to a port-group using VLAN 4095 using guest tagging.
- The broken setup has one pNIC as uplink for each of two vSwitches, the guest connected to a port-group using VLAN 4095 using guest tagging to both vSwitches.
If moving packets from eth0.1 to eth0.100 everything seems normal, but from eth0.100 to eth1.200 (i.e. across virtual adapters) then I get random duplicate TCP ACKs and it ends in garbled data and a TCP reset.
It if fully and easily reproducible. Changing the adapters to e1000/e1000e does not show the same problems.