[E1000-devel] Intel 2598EB 10-Gigabit AT dropped rx packet

[Mirko C. <mir...@ge...>]

Dear all,

I'm new to this list, if this isn't the right place to post this issue, 
please let me know.

I am testing two Intel 82598EB 10-Gigabit AT network card mounted on two
Dell R710 server with Red Hat Enterprise Linux 5.4 installed. The two
NICs are directly connected each other (no switch in between).   I am
experiencing a problem on the receive side of the connection: I'm sending
an UDP stream after having disabled flow control on each card, and about
the 60% of the transmitted packets are dropped with a large number of
rx_missed_errors. This is what ifconfig and ethtool -S show:

[root@grids2]# ethtool -S eth0
NIC statistics:
      rx_packets: 1761794
      tx_packets: 32
      rx_bytes: 14748667944
      tx_bytes: 5746
      lsc_int: 2
      tx_busy: 0
      non_eop_descs: 6510828
      rx_errors: 0
      tx_errors: 0
      rx_dropped: 0
      tx_dropped: 0
      multicast: 2
      broadcast: 2
      rx_no_buffer_count: 0
      collisions: 0
      rx_over_errors: 0
      rx_crc_errors: 0
      rx_frame_errors: 0
      rx_fifo_errors: 0
      rx_missed_errors: 1148382
      tx_aborted_errors: 0
      tx_carrier_errors: 0
      tx_fifo_errors: 0
      tx_heartbeat_errors: 0
      tx_timeout_count: 0
      tx_restart_queue: 0
      rx_long_length_errors: 0
      rx_short_length_errors: 0
      tx_tcp4_seg_ctxt: 0
      tx_tcp6_seg_ctxt: 0
      tx_flow_control_xon: 0
      rx_flow_control_xon: 0
      tx_flow_control_xoff: 0
      rx_flow_control_xoff: 0
      rx_csum_offload_good: 28
      rx_csum_offload_errors: 0
      tx_csum_offload_ctxt: 11
      low_latency_interrupt: 0
      alloc_rx_page_failed: 0
      alloc_rx_buff_failed: 0
      lro_aggregated: 8
      lro_flushed: 8
      lro_recycled: 0
      rx_no_dma_resources: 0
      hw_rsc_count: 0
      rx_flm: 0
      fdir_match: 0
      fdir_miss: 0
      tx_queue_0_packets: 32
      tx_queue_0_bytes: 5746
      rx_queue_0_packets: 2
      rx_queue_0_bytes: 120
      rx_queue_1_packets: 2
      rx_queue_1_bytes: 120
      rx_queue_2_packets: 1761762
      rx_queue_2_bytes: 14748664800
      rx_queue_3_packets: 0
      rx_queue_3_bytes: 0
      rx_queue_4_packets: 0
      rx_queue_4_bytes: 0
      rx_queue_5_packets: 14
      rx_queue_5_bytes: 1452
      rx_queue_6_packets: 7
      rx_queue_6_bytes: 726
      rx_queue_7_packets: 7
      rx_queue_7_bytes: 726

[root@grids2]# ifconfig eth0

eth0      Link encap:Ethernet  HWaddr 00:1B:21:4B:C8:BF
           inet addr:10.100.100.2  Bcast:10.100.100.255  Mask:255.255.255.0
           inet6 addr: fe80::21b:21ff:fe4b:c8bf/64 Scope:Link
           UP BROADCAST RUNNING MULTICAST  MTU:9000  Metric:1
           RX packets:1761795 errors:0 dropped:1148382 overruns:0 frame:0
           TX packets:39 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:1000
           RX bytes:14748668004 (13.7 GiB)  TX bytes:9332 (9.1 KiB)

Each interface has the MTU set to 9000, the driver is the one provided
by Dell support (ixgbe 2.0.44.14-NAPI) with Rx multiqueue enabled and
InterruptThrottleRate set to 8000 (but I tried 16000 and 32000 too).
Also I tried to set FdirPballoc parameter to 0, 1 and 2 with no success.
Single CPU load is no more than 10% with 8000 interrupt/s:

[root@grids2 ~]# mpstat -P ALL 2

05:16:43 PM  CPU   %user   %nice    %sys %iowait    %irq   %soft  %steal 
%idle    intr/s
05:16:45 PM  all    0.00    0.00    0.06    0.00    0.06    1.19    0.00 
    98.69   9018.00
05:16:45 PM    0    0.00    0.00    0.00    0.00    0.00    0.00    0.00 
   100.00   1002.50
05:16:45 PM    1    0.00    0.00    0.00    0.00    0.00    0.00    0.00 
   100.00      1.50
05:16:45 PM    2    0.00    0.00    0.50    0.00    0.50    9.45    0.00 
    89.55   8007.00
05:16:45 PM    3    0.00    0.00    0.00    0.00    0.00    0.00    0.00 
   100.00      1.00
05:16:45 PM    4    0.00    0.00    0.00    0.00    0.00    0.00    0.00 
   100.00      1.00
05:16:45 PM    5    0.00    0.00    0.00    0.00    0.00    0.00    0.00 
   100.00      3.00
05:16:45 PM    6    0.00    0.00    0.00    0.00    0.00    0.00    0.00 
   100.00      1.00
05:16:45 PM    7    0.00    0.00    0.00    0.00    0.00    0.00    0.00 
   100.00      1.00

I tried to configure smp affinity to assign the IRQ of each
rx-queue to a different CPU, with no effect on the amount of dropped
packets. I also verified that if I connect the NIC on a 10GE switch and
send more than one stream from different servers, the card uses
different rx-queues but the problem still exists.

I modified the /etc/sysctl.conf file parameters adding:

net.core.wmem_max = 16777216
net.core.wmem_default = 8388608
net.core.rmem_max = 16777216
net.core.rmem_default = 8388608
net.ipv4.tcp_rmem = 4096 262144 16777216
net.ipv4.tcp_wmem = 4096 262144 16777216
net.ipv4.tcp_mem = 4096 8388608 16777216
net.core.optmem_max = 524288
net.core.netdev_max_backlog = 200000




Some other details:

- CPU: dual quad core Intel L5520  @ 2.27GHz (Hyper-Threading disabled)
- RAM: 24 GB (12 X 2GB)
- Slot PCI: PCIe gen.2 x8
- Kernel linux: 2.6.18-128.1.1.el5 and 2.6.18-164.2.1.el5

Any suggestions?
Thank you very much in advance.

Mirko Corosu

-- 

--------------------------------------------------------------------------
Mirko Corosu
Network and system administrator
Computing Center
Istituto Nazionale Fisica Nucleare
Via Dodecaneso 33
16146 Genova, Italy
http://www.ge.infn.it
--------------------------------------------------------------------------