#413 IGB: Bad performance when using RX/TX VLAN hardware offload on RH6.x (with in-kernel & standalone [5.1.2] drivers)

closed
None
in-kernel_driver
1
2015-03-11
2014-05-22
Pablo Ruiz
No

Hi,

We have a few machines with both intel 82576 & 82575EB cards, and using CentOS 6.5 as distro. We have noticed a performance degradation under high bandwith (bidirectional) traffic, using in-kernel driver (kernel 2.6.32-358.14.1.el6.x86_64 with shipped igb v4.0.1) and standalone (v5.1.2) drivers.
However, the problem is much worst when using shipped driver than with the standalone.

When performing bi-directional bandwith tests with iperf, one side (usally TX) performs very poorly. RX side reaches nearly wirespeed, while TX performs a ~10% the speed of RX (using centos driver), or at ~45% the speed of RX (with standalone 5.1.2 [and 5.0.6]).

However, this performance issue is only reproducible when using vlan tagged iface. When using the native vlan interface, speed on both directions is nearly wirespeed.

Here is the example output from iperf a bidirectional tests against failing server (with igb v5.1.2), with iperf running in server mode (iperf -s -p 212), and iperf as client on a different machine (using a different network card), in both cases the traffic goes thru a VLAN tagged interface (eth0.210):

iperf -c 172.21.210.132 -p 212 -L 213 -d -t 15


Server listening on TCP port 213
TCP window size: 85.3 KByte (default)



Client connecting to 172.21.210.132, TCP port 212
TCP window size: 128 KByte (default)


[ 5] local 172.21.210.101 port 36425 connected with 172.21.210.132 port 212
[ 4] local 172.21.210.101 port 213 connected with 172.21.210.132 port 46976
[ ID] Interval Transfer Bandwidth
[ 5] 0.0-15.0 sec 1.61 GBytes 920 Mbits/sec
[ 4] 0.0-15.0 sec 858 MBytes 479 Mbits/sec

When using CentOS shipped kernel (v4.0.1), as said, the results are much worse:

iperf -c 172.21.210.131 -p 212 -L 213 -d -t 15


Server listening on TCP port 213
TCP window size: 85.3 KByte (default)



Client connecting to 172.21.210.131, TCP port 212
TCP window size: 167 KByte (default)


[ 5] local 172.21.210.101 port 41663 connected with 172.21.210.131 port 212
[ 4] local 172.21.210.101 port 213 connected with 172.21.210.131 port 40365
[ ID] Interval Transfer Bandwidth
[ 5] 0.0-15.0 sec 80.6 MBytes 45.0 Mbits/sec
[ 4] 0.0-15.0 sec 1.64 GBytes 937 Mbits/sec

However, this same tests on this same servers, using the default/native vlan interface (eth0):

iperf -c 172.21.1.131 -p 212 -L 213 -d -t 15


Server listening on TCP port 213
TCP window size: 85.3 KByte (default)



Client connecting to 172.21.1.131, TCP port 212
TCP window size: 181 KByte (default)


[ 5] local 172.21.1.101 port 41649 connected with 172.21.1.131 port 212
[ 4] local 172.21.1.101 port 213 connected with 172.21.1.131 port 40362
[ ID] Interval Transfer Bandwidth
[ 5] 0.0-15.0 sec 1.59 GBytes 911 Mbits/sec
[ 4] 0.0-15.0 sec 1.58 GBytes 904 Mbits/sec

After a thoroughly diagnosis, the culprit seems to be hardware vlan offload. As when disabling such feature from the driver, everything works as expected (albeit using a bit more CPU/Ints ;)).

However after digging around with all the available intel igb & kernel settings, if found disable rxvlan/txvlan accel using ethtool on CentOS/Redhat 6.x is not possible due to rh's kernel (which is missing some features available at modern 3.x kernels).

So, as a workaround, I've patched (attached to this issue) driver v5.1.2 adding a new module option by which you can override hw vlan offload.

1 Attachments

Discussion

  • Todd Fujinaka

    Todd Fujinaka - 2014-06-20
    • assigned_to: Todd Fujinaka
     
  • Pablo Ruiz

    Pablo Ruiz - 2014-06-21

    As an update, disabling vlan hardware accel, did not fix the issue, it allieviated it on some situations, but the issue still persists. :(

    I've tested again with newer released rh/centos kernels: 2.6.32-431.17.1.el6 & 2.6.32-431.20.3.el6 (both x64), but still failing.

     
  • Todd Fujinaka

    Todd Fujinaka - 2015-03-11
    • status: open --> closed
     
  • Todd Fujinaka

    Todd Fujinaka - 2015-03-11

    I would file this issue with RedHat if you're still seeing problems.

     

Get latest updates about Open Source Projects, Conferences and News.

Sign up for the SourceForge newsletter:





No, thanks