Menu
▾
▴
Re: [E1000-devel] Intel 2598EB 10-Gigabit AT dropped rx packet
|
From: Mirko C. <mir...@ge...> - 2010-03-31 23:28:27
|
Sorry for the late, I missed the last message. > Mirko, did you ever try sending two UDP streams? > Yes, I tried also more than one UDP stream with no luck. > > rx_missed_error is the number of packets that have been dropped due to > no HOST memory buffers available. The driver provides host memory to > the controller via the driver's interrupt routine. > > In your case due to your single threaded workload you're limited by what > a single cpu can process. The output of mpstat shows very low cpu load during the tests. I contacted Dell support to investigate any possible bottleneck due to the internal server architecture. > > On Thu, 2010-02-25 at 00:56 -0800, Mirko Corosu wrote: >> Hi Donald, >> >>> Once again to clarify your concern is: >>> >>> TX system -> RX system (TCP) 9.5 Gb/s >>> TX system -> RX system (UDP / FC off) 5.6 Gb/s >> >> Sorry... not exactly, but I guess I found the reason why we can't >> understand each other. When I say: "on TX side I get 9.5 Gb/s" I mean >> that I measure the network traffic through the interface on the TX >> system (I use dstat for that). >> >> I try to clarify further my numbers: >> >> All tests are from TX system to RX system >> >> UDP (no FC): >> through eth0 on TX system: 9.5 Gb/s >> through eth0 on RX system: 5.6 Gb/s >> >> UDP (FC): >> through eth0 on TX system 5.6 Gb/s >> through eth0 on RX system 5.6 Gb/s >> >> TCP (no FC): >> through eth0 on TX system 5.6 Gb/s >> through eth0 on RX system 5.6 Gb/s >> >> TCP (FC): >> through eth0 on TX system 5.6 Gb/s >> through eth0 on RX system 5.6 Gb/s >> >> FC does not improve my throughput (but of course eliminates >> rx_missed_errors). >> >>> As for the meaning of the rx_missed_errors I can at least help you with that. I imagined you looked at the code but it's the summation of an array of registers called MPC(), one for each receive FIFO. It's incremented when Packets are missed due to insufficient space to store that packet. This might be due to bandwidth issue with the bus IO (which is why I was asking about the slot you have the NIC's in) or too few buffers allocated. If I remember correctly you already tried that. Or in our case a transmitter blasting out packets as faster than we can receive them >> Ok. As I wrote before this is my network buffer configuration: >> >> net.core.wmem_max = 16777216 >> net.core.wmem_default = 8388608 >> net.core.rmem_max = 16777216 >> net.core.rmem_default = 8388608 >> net.ipv4.tcp_rmem = 4096 262144 16777216 >> net.ipv4.tcp_wmem = 4096 262144 16777216 >> net.ipv4.tcp_mem = 4096 8388608 16777216 >> net.core.optmem_max = 524288 >> net.core.netdev_max_backlog = 200000 >> >> AFAIK, these are the only parameters I can set. >> >> Thanks for your patience :-) >> >> Mirko >> >>>> -----Original Message----- >>>> From: Mirko Corosu [mailto:mir...@ge...] >>>> Sent: Wednesday, February 24, 2010 2:56 AM >>>> To: E10...@li... >>>> Subject: Re: [E1000-devel] Intel 2598EB 10-Gigabit AT dropped rx packet >>>> >>>> Hi Donald, >>>> >>>> >>>>> One thing I did notice was that the Receive side has 8 rx queues while >>>> the Transmit has 16 rx queues. You mentioned these were identical machines >>>> could Hyper-Threading be off on the Receiving machine? >>>> Yes, it was my fault. I forgot to disable hyperthreading on TX server. >>>> Now it is disabled, but the problem still remains. >>>> >>>>> Also I'm still a little confused about how your test is set up. If you >>>> have four netperf's running on, as you called it, your Tx machine and your >>>> Rx is only running netserver then I would only expect heavy traffic from Tx >>>> -> Rx. Or am I missing some thing here? >>>> I try to summarize my tests. >>>> >>>> For now I am testing only mono-directional TCP and UDP stream (from TX >>>> server to RX server directly ). What I cannot understand is the low >>>> throughput that I'm getting. The TCP tests shows a throughput of only >>>> 5.6 Gb/s. During UDP tests, with flow control disabled, I measured on TX >>>> side a throughput of about 9.5 Gb/s (so the card can send at wire speed) >>>> but on RX side I see only 5.6 Gb/s. I attribute this problem to the high >>>> number of rx_missed_errors but I cannot figure out what is the real >>>> cause (I don't know even what rx_missed_errors actually mean....). >>>> >>>> Thanks >>>> >>>> Mirko >>>> >>>>>> -----Original Message----- >>>>>> From: Mirko Corosu [mailto:mir...@ge...] >>>>>> Sent: Tuesday, February 23, 2010 3:30 AM >>>>>> To: Skidmore, Donald C >>>>>> Cc: E10...@li... >>>>>> Subject: Re: [E1000-devel] Intel 2598EB 10-Gigabit AT dropped rx packet >>>>>> >>>>>> Hi Donald, >>>>>> >>>>>> Ok, sorry for the lack of informations: I'm not used to interact with >>>>>> developers. :-) >>>>>> >>>>>> >>>>>>> How were you using netperf in your tests? What tests (I assume >>>>>> [UDP|TCP]_STREAM) and how many instances of netperf were running each >>>> side? >>>>>> I'm running four instances of UDP_STREAM or TCP_STREAM tests on the tx >>>>>> side, with different packet sizes (from 1000 to 9000 byte) and a single >>>>>> netserver on the rx side. >>>>>> >>>>>>> From the spec I got off the web on the R710 I can see they have either >>>> (2 >>>>>> PCIe x8 and 2 PCIe x4) or (1 PCIe x16 and 2 PCIe x4 ). I was just >>>>>> concerned you might be plugged into one of the x4 slots. >>>>>> >>>>>> I have 2 PCIe x8 on the so called "Riser 2" and 2 PCIe x4 on the "Riser >>>>>> 1". On each server the Intel cards are plugged into the Riser 2 slots. >>>>>> >>>>>>>>> - Did you look at the dmesg for anything relative? >>>>>>>> I see no error or warning messages on both side. >>>>>>> It's not just error or warning messages I was interested in, although >>>>>> they would be interesting too. :) We also printk a message that >>>> displays >>>>>> the type of PCIe link we have. This would answer the question I posed >>>>>> above about what slot the cards were in. >>>>>> On rx side: >>>>>> >>>>>> [root@grids2 ~]# dmesg |grep ixgbe >>>>>> ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version >>>>>> 2.0.44.14-NAPI >>>>>> ixgbe: 0000:07:00.0: ixgbe_init_interrupt_scheme: Multiqueue Enabled: Rx >>>>>> Queue count = 8, Tx Queue count = 1 >>>>>> ixgbe: eth0: ixgbe_probe: (PCI Express:2.5Gb/s:Width x8) >>>> 00:1b:21:4b:c8:bf >>>>>> ixgbe: eth0: ixgbe_probe: MAC: 1, PHY: 2, PBA No: d79893-017 >>>>>> ixgbe: eth0: ixgbe_probe: Internal LRO is enabled >>>>>> ixgbe: eth0: ixgbe_probe: Intel(R) 10 Gigabit Network Connection >>>>>> ixgbe: eth0: ixgbe_change_mtu: changing MTU from 1500 to 9000 >>>>>> ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Up 10 Gbps, Flow Control: >>>>>> RX/TX >>>>>> ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Up 10 Gbps, Flow Control: >>>>>> None >>>>>> ixgbe: eth0: ixgbe_remove: complete >>>>>> ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version >>>>>> 2.0.44.14-NAPI >>>>>> ixgbe: 0000:07:00.0: ixgbe_init_interrupt_scheme: Multiqueue Enabled: Rx >>>>>> Queue count = 8, Tx Queue count = 1 >>>>>> ixgbe: eth0: ixgbe_probe: (PCI Express:2.5Gb/s:Width x8) >>>> 00:1b:21:4b:c8:bf >>>>>> ixgbe: eth0: ixgbe_probe: MAC: 1, PHY: 2, PBA No: d79893-017 >>>>>> ixgbe: eth0: ixgbe_probe: Internal LRO is enabled >>>>>> ixgbe: eth0: ixgbe_probe: Intel(R) 10 Gigabit Network Connection >>>>>> ixgbe: eth0: ixgbe_change_mtu: changing MTU from 1500 to 9000 >>>>>> ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Up 10 Gbps, Flow Control: >>>>>> RX/TX >>>>>> ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Up 10 Gbps, Flow Control: >>>>>> None >>>>>> ixgbe: eth0: ixgbe_remove: complete >>>>>> ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version >>>>>> 2.0.44.14-NAPI >>>>>> ixgbe: 0000:07:00.0: ixgbe_init_interrupt_scheme: Multiqueue Enabled: Rx >>>>>> Queue count = 8, Tx Queue count = 1 >>>>>> ixgbe: eth0: ixgbe_probe: (PCI Express:2.5Gb/s:Width x8) >>>> 00:1b:21:4b:c8:bf >>>>>> ixgbe: eth0: ixgbe_probe: MAC: 1, PHY: 2, PBA No: d79893-017 >>>>>> ixgbe: eth0: ixgbe_probe: Internal LRO is enabled >>>>>> ixgbe: eth0: ixgbe_probe: Intel(R) 10 Gigabit Network Connection >>>>>> ixgbe: eth0: ixgbe_change_mtu: changing MTU from 1500 to 9000 >>>>>> ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Up 10 Gbps, Flow Control: >>>>>> RX/TX >>>>>> ixgbe: eth0: ixgbe_remove: complete >>>>>> ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version >>>>>> 2.0.44.14-NAPI >>>>>> ixgbe: 0000:07:00.0: ixgbe_init_interrupt_scheme: Multiqueue Enabled: Rx >>>>>> Queue count = 8, Tx Queue count = 1 >>>>>> ixgbe: eth0: ixgbe_probe: (PCI Express:2.5Gb/s:Width x8) >>>> 00:1b:21:4b:c8:bf >>>>>> ixgbe: eth0: ixgbe_probe: MAC: 1, PHY: 2, PBA No: d79893-017 >>>>>> ixgbe: eth0: ixgbe_probe: Internal LRO is enabled >>>>>> ixgbe: eth0: ixgbe_probe: Intel(R) 10 Gigabit Network Connection >>>>>> ixgbe: eth0: ixgbe_change_mtu: changing MTU from 1500 to 9000 >>>>>> ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Up 10 Gbps, Flow Control: >>>>>> RX/TX >>>>>> ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Up 10 Gbps, Flow Control: >>>>>> None >>>>>> ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Up 10 Gbps, Flow Control: >>>>>> RX/TX >>>>>> ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Down >>>>>> ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Up 10 Gbps, Flow Control: >>>>>> RX/TX >>>>>> ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Down >>>>>> ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Up 10 Gbps, Flow Control: >>>>>> RX/TX >>>>>> ixgbe: eth0: ixgbe_remove: complete >>>>>> ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version >>>>>> 2.0.44.14-NAPI >>>>>> ixgbe: 0000:07:00.0: ixgbe_init_interrupt_scheme: Multiqueue Enabled: Rx >>>>>> Queue count = 8, Tx Queue count = 1 >>>>>> ixgbe: eth0: ixgbe_probe: (PCI Express:2.5Gb/s:Width x8) >>>> 00:1b:21:4b:c8:bf >>>>>> ixgbe: eth0: ixgbe_probe: MAC: 1, PHY: 2, PBA No: d79893-017 >>>>>> ixgbe: eth0: ixgbe_probe: Internal LRO is enabled >>>>>> ixgbe: eth0: ixgbe_probe: Intel(R) 10 Gigabit Network Connection >>>>>> ixgbe: eth0: ixgbe_change_mtu: changing MTU from 1500 to 9000 >>>>>> ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Up 10 Gbps, Flow Control: >>>>>> RX/TX >>>>>> ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Down >>>>>> ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Up 10 Gbps, Flow Control: >>>>>> RX/TX >>>>>> ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Down >>>>>> ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Up 10 Gbps, Flow Control: >>>>>> RX/TX >>>>>> ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Down >>>>>> ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Up 10 Gbps, Flow Control: >>>>>> RX/TX >>>>>> ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Down >>>>>> ixgbe: eth0: ixgbe_remove: complete >>>>>> ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version >>>>>> 2.0.44.14-NAPI >>>>>> ixgbe: 0000:07:00.0: ixgbe_init_interrupt_scheme: Multiqueue Enabled: Rx >>>>>> Queue count = 8, Tx Queue count = 1 >>>>>> ixgbe: eth0: ixgbe_probe: (PCI Express:2.5Gb/s:Width x8) >>>> 00:1b:21:4b:c8:bf >>>>>> ixgbe: eth0: ixgbe_probe: MAC: 1, PHY: 2, PBA No: d79893-017 >>>>>> ixgbe: eth0: ixgbe_probe: Internal LRO is enabled >>>>>> ixgbe: eth0: ixgbe_probe: Intel(R) 10 Gigabit Network Connection >>>>>> ixgbe: eth0: ixgbe_change_mtu: changing MTU from 1500 to 9000 >>>>>> ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Up 10 Gbps, Flow Control: >>>>>> RX/TX >>>>>> >>>>>> >>>>>> >>>>>> On tx side: >>>>>> >>>>>> [root@client20 ~]# dmesg |grep ixgbe >>>>>> ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version >>>>>> 2.0.44.14-NAPI >>>>>> ixgbe: 0000:07:00.0: ixgbe_init_interrupt_scheme: Multiqueue Enabled: Rx >>>>>> Queue count = 16, Tx Queue count = 1 >>>>>> ixgbe: eth0: ixgbe_probe: No DCA provider found. Please start ioatdma >>>>>> for DCA functionality. >>>>>> ixgbe: eth0: ixgbe_probe: (PCI Express:2.5Gb/s:Width x8) >>>> 00:1b:21:4b:c8:e0 >>>>>> ixgbe: eth0: ixgbe_probe: MAC: 1, PHY: 2, PBA No: d79893-017 >>>>>> ixgbe: eth0: ixgbe_probe: Internal LRO is enabled >>>>>> ixgbe: eth0: ixgbe_probe: Intel(R) 10 Gigabit Network Connection >>>>>> ixgbe: eth0: ixgbe_change_mtu: changing MTU from 1500 to 9000 >>>>>> ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Up 10 Gbps, Flow Control: >>>>>> None >>>>>> ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Up 10 Gbps, Flow Control: >>>>>> RX/TX >>>>>> ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Down >>>>>> ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Up 10 Gbps, Flow Control: >>>>>> RX/TX >>>>>> ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Down >>>>>> ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Up 10 Gbps, Flow Control: >>>>>> RX/TX >>>>>> ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Down >>>>>> ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Up 10 Gbps, Flow Control: >>>>>> RX/TX >>>>>> ixgbe: eth0: ixgbe_remove: complete >>>>>> ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version >>>>>> 2.0.44.14-NAPI >>>>>> ixgbe: 0000:07:00.0: ixgbe_init_interrupt_scheme: Multiqueue Enabled: Rx >>>>>> Queue count = 16, Tx Queue count = 1 >>>>>> ixgbe: eth0: ixgbe_probe: No DCA provider found. Please start ioatdma >>>>>> for DCA functionality. >>>>>> ixgbe: eth0: ixgbe_probe: (PCI Express:2.5Gb/s:Width x8) >>>> 00:1b:21:4b:c8:e0 >>>>>> ixgbe: eth0: ixgbe_probe: MAC: 1, PHY: 2, PBA No: d79893-017 >>>>>> ixgbe: eth0: ixgbe_probe: Internal LRO is enabled >>>>>> ixgbe: eth0: ixgbe_probe: Intel(R) 10 Gigabit Network Connection >>>>>> ixgbe: eth0: ixgbe_change_mtu: changing MTU from 1500 to 9000 >>>>>> ixgbe: eth0: ixgbe_remove: complete >>>>>> ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version >>>>>> 2.0.44.14-NAPI >>>>>> ixgbe: 0000:07:00.0: ixgbe_init_interrupt_scheme: Multiqueue Enabled: Rx >>>>>> Queue count = 16, Tx Queue count = 1 >>>>>> ixgbe: eth0: ixgbe_probe: No DCA provider found. Please start ioatdma >>>>>> for DCA functionality. >>>>>> ixgbe: eth0: ixgbe_probe: (PCI Express:2.5Gb/s:Width x8) >>>> 00:1b:21:4b:c8:e0 >>>>>> ixgbe: eth0: ixgbe_probe: MAC: 1, PHY: 2, PBA No: d79893-017 >>>>>> ixgbe: eth0: ixgbe_probe: Internal LRO is enabled >>>>>> ixgbe: eth0: ixgbe_probe: Intel(R) 10 Gigabit Network Connection >>>>>> ixgbe: eth0: ixgbe_change_mtu: changing MTU from 1500 to 9000 >>>>>> ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Up 10 Gbps, Flow Control: >>>>>> RX/TX >>>>>> ixgbe: eth0: ixgbe_remove: complete >>>>>> ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version >>>>>> 2.0.44.14-NAPI >>>>>> ixgbe: 0000:07:00.0: ixgbe_init_interrupt_scheme: Multiqueue Enabled: Rx >>>>>> Queue count = 16, Tx Queue count = 1 >>>>>> ixgbe: eth0: ixgbe_probe: No DCA provider found. Please start ioatdma >>>>>> for DCA functionality. >>>>>> ixgbe: eth0: ixgbe_probe: (PCI Express:2.5Gb/s:Width x8) >>>> 00:1b:21:4b:c8:e0 >>>>>> ixgbe: eth0: ixgbe_probe: MAC: 1, PHY: 2, PBA No: d79893-017 >>>>>> ixgbe: eth0: ixgbe_probe: Internal LRO is enabled >>>>>> ixgbe: eth0: ixgbe_probe: Intel(R) 10 Gigabit Network Connection >>>>>> ixgbe: eth0: ixgbe_change_mtu: changing MTU from 1500 to 9000 >>>>>> ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Up 10 Gbps, Flow Control: >>>>>> RX/TX >>>>>> >>>>>> >>>>>> >>>>>>> All that said we would expect to see rx_missed_errors with flow control >>>>>> disabled. Since nothing would be stopping packets from overrunning the >>>>>> buffer. Your test seems to be agreeing with this as the number of >>>>>> rx_missed_errors goes down with TCP (like you said) most likely due to >>>> TCP >>>>>> sliding window. >>>>>>> You don't see this issue with flow control enabled, right? >>>>>> Right, but the throughput remains about 5.6 Gb/s and a can't figure out >>>>>> where the bottleneck could be.... >>>>>> >>>>>> Thanks a lot >>>>>> >>>>>> Mirko >>>>>> >>>>>>>> -----Original Message----- >>>>>>>> From: Mirko Corosu [mailto:mir...@ge...] >>>>>>>> Sent: Saturday, February 20, 2010 10:50 AM >>>>>>>> To: Skidmore, Donald C >>>>>>>> Cc: E10...@li... >>>>>>>> Subject: Re: [E1000-devel] Intel 2598EB 10-Gigabit AT dropped rx >>>> packet >>>>>>>> Hi Donald, >>>>>>>> >>>>>>>> Thank you for your reply. >>>>>>>> >>>>>>>>> I have a couple additional questions >>>>>>>>> - What were you running to do this test? (i.e. netperf, pktgen?) >>>>>>>> netperf-2.4.2-1.el5.rf >>>>>>>> >>>>>>>>> - What throughput were you seeing? >>>>>>>> On tx side i see almost 10Gb/s, on rx side I see about 5.6 Gb/s >>>>>>>> >>>>>>>>> - Are both ports plugged into x8 PCIe slots >>>>>>>> Yes, both servers are Dell R710 with the same hardware configuration >>>>>>>> >>>>>>>>> - Did you look at the dmesg for anything relative? >>>>>>>> I see no error or warning messages on both side. >>>>>>>> >>>>>>>>> - Do you see this issue only with UDP? >>>>>>>> If I send a TCP stream I see on both side a throughput of about 5.54 >>>>>>>> Gb/s and less rx_missed_errors on rx side: >>>>>>>> >>>>>>>> NIC statistics: >>>>>>>> rx_packets: 1768101 >>>>>>>> tx_packets: 235384 >>>>>>>> rx_bytes: 15937257320 >>>>>>>> tx_bytes: 15704254 >>>>>>>> lsc_int: 2 >>>>>>>> tx_busy: 0 >>>>>>>> non_eop_descs: 7072222 >>>>>>>> rx_errors: 0 >>>>>>>> tx_errors: 0 >>>>>>>> rx_dropped: 0 >>>>>>>> tx_dropped: 0 >>>>>>>> multicast: 1 >>>>>>>> broadcast: 0 >>>>>>>> rx_no_buffer_count: 0 >>>>>>>> collisions: 0 >>>>>>>> rx_over_errors: 0 >>>>>>>> rx_crc_errors: 0 >>>>>>>> rx_frame_errors: 0 >>>>>>>> rx_fifo_errors: 0 >>>>>>>> rx_missed_errors: 1539 >>>>>>>> tx_aborted_errors: 0 >>>>>>>> tx_carrier_errors: 0 >>>>>>>> tx_fifo_errors: 0 >>>>>>>> tx_heartbeat_errors: 0 >>>>>>>> tx_timeout_count: 0 >>>>>>>> tx_restart_queue: 0 >>>>>>>> rx_long_length_errors: 0 >>>>>>>> rx_short_length_errors: 0 >>>>>>>> tx_tcp4_seg_ctxt: 0 >>>>>>>> tx_tcp6_seg_ctxt: 0 >>>>>>>> tx_flow_control_xon: 0 >>>>>>>> rx_flow_control_xon: 0 >>>>>>>> tx_flow_control_xoff: 0 >>>>>>>> rx_flow_control_xoff: 0 >>>>>>>> rx_csum_offload_good: 1768098 >>>>>>>> rx_csum_offload_errors: 0 >>>>>>>> tx_csum_offload_ctxt: 11 >>>>>>>> low_latency_interrupt: 0 >>>>>>>> alloc_rx_page_failed: 0 >>>>>>>> alloc_rx_buff_failed: 0 >>>>>>>> lro_aggregated: 1760417 >>>>>>>> lro_flushed: 375277 >>>>>>>> lro_recycled: 1012791 >>>>>>>> rx_no_dma_resources: 0 >>>>>>>> hw_rsc_count: 0 >>>>>>>> rx_flm: 0 >>>>>>>> fdir_match: 0 >>>>>>>> fdir_miss: 0 >>>>>>>> tx_queue_0_packets: 235384 >>>>>>>> tx_queue_0_bytes: 15704254 >>>>>>>> rx_queue_0_packets: 2 >>>>>>>> rx_queue_0_bytes: 120 >>>>>>>> rx_queue_1_packets: 8 >>>>>>>> rx_queue_1_bytes: 786 >>>>>>>> rx_queue_2_packets: 301752 >>>>>>>> rx_queue_2_bytes: 2719959436 >>>>>>>> rx_queue_3_packets: 562919 >>>>>>>> rx_queue_3_bytes: 5074118826 >>>>>>>> rx_queue_4_packets: 7 >>>>>>>> rx_queue_4_bytes: 726 >>>>>>>> rx_queue_5_packets: 15 >>>>>>>> rx_queue_5_bytes: 1526 >>>>>>>> rx_queue_6_packets: 313901 >>>>>>>> rx_queue_6_bytes: 2829476778 >>>>>>>> rx_queue_7_packets: 589497 >>>>>>>> rx_queue_7_bytes: 5313699122 >>>>>>>> >>>>>>>> I guess it's because of TCP flow control. >>>>>>>> >>>>>>>> I remember that the same behavior was triggered by enabling flow >>>> control >>>>>>>> on both cards (ethtool -A eth0 rx on tx on autoconf on) but I have to >>>>>>>> wait till Monday to perform another run to test that configuration. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> Mirko >>>>>>>> >>>>>>>>> What I'm wondering is if this NIC was receiving faster than it could >>>>>>>> process them. >>>>>>>>> Thanks, >>>>>>>>> -Don Skidmore <Don...@in...> >>>>>>>>> >>>>>>>>> >>>>>>>>>> -----Original Message----- >>>>>>>>>> From: Mirko Corosu [mailto:mir...@ge...] >>>>>>>>>> Sent: Friday, February 19, 2010 5:30 AM >>>>>>>>>> To: E10...@li... >>>>>>>>>> Subject: [E1000-devel] Intel 2598EB 10-Gigabit AT dropped rx packet >>>>>>>>>> >>>>>>>>>> Dear all, >>>>>>>>>> >>>>>>>>>> I'm new to this list, if this isn't the right place to post this >>>>>> issue, >>>>>>>>>> please let me know. >>>>>>>>>> >>>>>>>>>> I am testing two Intel 82598EB 10-Gigabit AT network card mounted on >>>>>> two >>>>>>>>>> Dell R710 server with Red Hat Enterprise Linux 5.4 installed. The >>>> two >>>>>>>>>> NICs are directly connected each other (no switch in between). I >>>> am >>>>>>>>>> experiencing a problem on the receive side of the connection: I'm >>>>>>>> sending >>>>>>>>>> an UDP stream after having disabled flow control on each card, and >>>>>> about >>>>>>>>>> the 60% of the transmitted packets are dropped with a large number >>>> of >>>>>>>>>> rx_missed_errors. This is what ifconfig and ethtool -S show: >>>>>>>>>> >>>>>>>>>> [root@grids2]# ethtool -S eth0 >>>>>>>>>> NIC statistics: >>>>>>>>>> rx_packets: 1761794 >>>>>>>>>> tx_packets: 32 >>>>>>>>>> rx_bytes: 14748667944 >>>>>>>>>> tx_bytes: 5746 >>>>>>>>>> lsc_int: 2 >>>>>>>>>> tx_busy: 0 >>>>>>>>>> non_eop_descs: 6510828 >>>>>>>>>> rx_errors: 0 >>>>>>>>>> tx_errors: 0 >>>>>>>>>> rx_dropped: 0 >>>>>>>>>> tx_dropped: 0 >>>>>>>>>> multicast: 2 >>>>>>>>>> broadcast: 2 >>>>>>>>>> rx_no_buffer_count: 0 >>>>>>>>>> collisions: 0 >>>>>>>>>> rx_over_errors: 0 >>>>>>>>>> rx_crc_errors: 0 >>>>>>>>>> rx_frame_errors: 0 >>>>>>>>>> rx_fifo_errors: 0 >>>>>>>>>> rx_missed_errors: 1148382 >>>>>>>>>> tx_aborted_errors: 0 >>>>>>>>>> tx_carrier_errors: 0 >>>>>>>>>> tx_fifo_errors: 0 >>>>>>>>>> tx_heartbeat_errors: 0 >>>>>>>>>> tx_timeout_count: 0 >>>>>>>>>> tx_restart_queue: 0 >>>>>>>>>> rx_long_length_errors: 0 >>>>>>>>>> rx_short_length_errors: 0 >>>>>>>>>> tx_tcp4_seg_ctxt: 0 >>>>>>>>>> tx_tcp6_seg_ctxt: 0 >>>>>>>>>> tx_flow_control_xon: 0 >>>>>>>>>> rx_flow_control_xon: 0 >>>>>>>>>> tx_flow_control_xoff: 0 >>>>>>>>>> rx_flow_control_xoff: 0 >>>>>>>>>> rx_csum_offload_good: 28 >>>>>>>>>> rx_csum_offload_errors: 0 >>>>>>>>>> tx_csum_offload_ctxt: 11 >>>>>>>>>> low_latency_interrupt: 0 >>>>>>>>>> alloc_rx_page_failed: 0 >>>>>>>>>> alloc_rx_buff_failed: 0 >>>>>>>>>> lro_aggregated: 8 >>>>>>>>>> lro_flushed: 8 >>>>>>>>>> lro_recycled: 0 >>>>>>>>>> rx_no_dma_resources: 0 >>>>>>>>>> hw_rsc_count: 0 >>>>>>>>>> rx_flm: 0 >>>>>>>>>> fdir_match: 0 >>>>>>>>>> fdir_miss: 0 >>>>>>>>>> tx_queue_0_packets: 32 >>>>>>>>>> tx_queue_0_bytes: 5746 >>>>>>>>>> rx_queue_0_packets: 2 >>>>>>>>>> rx_queue_0_bytes: 120 >>>>>>>>>> rx_queue_1_packets: 2 >>>>>>>>>> rx_queue_1_bytes: 120 >>>>>>>>>> rx_queue_2_packets: 1761762 >>>>>>>>>> rx_queue_2_bytes: 14748664800 >>>>>>>>>> rx_queue_3_packets: 0 >>>>>>>>>> rx_queue_3_bytes: 0 >>>>>>>>>> rx_queue_4_packets: 0 >>>>>>>>>> rx_queue_4_bytes: 0 >>>>>>>>>> rx_queue_5_packets: 14 >>>>>>>>>> rx_queue_5_bytes: 1452 >>>>>>>>>> rx_queue_6_packets: 7 >>>>>>>>>> rx_queue_6_bytes: 726 >>>>>>>>>> rx_queue_7_packets: 7 >>>>>>>>>> rx_queue_7_bytes: 726 >>>>>>>>>> >>>>>>>>>> [root@grids2]# ifconfig eth0 >>>>>>>>>> >>>>>>>>>> eth0 Link encap:Ethernet HWaddr 00:1B:21:4B:C8:BF >>>>>>>>>> inet addr:10.100.100.2 Bcast:10.100.100.255 >>>>>>>> Mask:255.255.255.0 >>>>>>>>>> inet6 addr: fe80::21b:21ff:fe4b:c8bf/64 Scope:Link >>>>>>>>>> UP BROADCAST RUNNING MULTICAST MTU:9000 Metric:1 >>>>>>>>>> RX packets:1761795 errors:0 dropped:1148382 overruns:0 >>>>>> frame:0 >>>>>>>>>> TX packets:39 errors:0 dropped:0 overruns:0 carrier:0 >>>>>>>>>> collisions:0 txqueuelen:1000 >>>>>>>>>> RX bytes:14748668004 (13.7 GiB) TX bytes:9332 (9.1 KiB) >>>>>>>>>> >>>>>>>>>> Each interface has the MTU set to 9000, the driver is the one >>>> provided >>>>>>>>>> by Dell support (ixgbe 2.0.44.14-NAPI) with Rx multiqueue enabled >>>> and >>>>>>>>>> InterruptThrottleRate set to 8000 (but I tried 16000 and 32000 too). >>>>>>>>>> Also I tried to set FdirPballoc parameter to 0, 1 and 2 with no >>>>>> success. >>>>>>>>>> Single CPU load is no more than 10% with 8000 interrupt/s: >>>>>>>>>> >>>>>>>>>> [root@grids2 ~]# mpstat -P ALL 2 >>>>>>>>>> >>>>>>>>>> 05:16:43 PM CPU %user %nice %sys %iowait %irq %soft >>>>>> %steal >>>>>>>>>> %idle intr/s >>>>>>>>>> 05:16:45 PM all 0.00 0.00 0.06 0.00 0.06 1.19 >>>>>> 0.00 >>>>>>>>>> 98.69 9018.00 >>>>>>>>>> 05:16:45 PM 0 0.00 0.00 0.00 0.00 0.00 0.00 >>>>>> 0.00 >>>>>>>>>> 100.00 1002.50 >>>>>>>>>> 05:16:45 PM 1 0.00 0.00 0.00 0.00 0.00 0.00 >>>>>> 0.00 >>>>>>>>>> 100.00 1.50 >>>>>>>>>> 05:16:45 PM 2 0.00 0.00 0.50 0.00 0.50 9.45 >>>>>> 0.00 >>>>>>>>>> 89.55 8007.00 >>>>>>>>>> 05:16:45 PM 3 0.00 0.00 0.00 0.00 0.00 0.00 >>>>>> 0.00 >>>>>>>>>> 100.00 1.00 >>>>>>>>>> 05:16:45 PM 4 0.00 0.00 0.00 0.00 0.00 0.00 >>>>>> 0.00 >>>>>>>>>> 100.00 1.00 >>>>>>>>>> 05:16:45 PM 5 0.00 0.00 0.00 0.00 0.00 0.00 >>>>>> 0.00 >>>>>>>>>> 100.00 3.00 >>>>>>>>>> 05:16:45 PM 6 0.00 0.00 0.00 0.00 0.00 0.00 >>>>>> 0.00 >>>>>>>>>> 100.00 1.00 >>>>>>>>>> 05:16:45 PM 7 0.00 0.00 0.00 0.00 0.00 0.00 >>>>>> 0.00 >>>>>>>>>> 100.00 1.00 >>>>>>>>>> >>>>>>>>>> I tried to configure smp affinity to assign the IRQ of each >>>>>>>>>> rx-queue to a different CPU, with no effect on the amount of dropped >>>>>>>>>> packets. I also verified that if I connect the NIC on a 10GE switch >>>>>> and >>>>>>>>>> send more than one stream from different servers, the card uses >>>>>>>>>> different rx-queues but the problem still exists. >>>>>>>>>> >>>>>>>>>> I modified the /etc/sysctl.conf file parameters adding: >>>>>>>>>> >>>>>>>>>> net.core.wmem_max = 16777216 >>>>>>>>>> net.core.wmem_default = 8388608 >>>>>>>>>> net.core.rmem_max = 16777216 >>>>>>>>>> net.core.rmem_default = 8388608 >>>>>>>>>> net.ipv4.tcp_rmem = 4096 262144 16777216 >>>>>>>>>> net.ipv4.tcp_wmem = 4096 262144 16777216 >>>>>>>>>> net.ipv4.tcp_mem = 4096 8388608 16777216 >>>>>>>>>> net.core.optmem_max = 524288 >>>>>>>>>> net.core.netdev_max_backlog = 200000 >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Some other details: >>>>>>>>>> >>>>>>>>>> - CPU: dual quad core Intel L5520 @ 2.27GHz (Hyper-Threading >>>>>> disabled) >>>>>>>>>> - RAM: 24 GB (12 X 2GB) >>>>>>>>>> - Slot PCI: PCIe gen.2 x8 >>>>>>>>>> - Kernel linux: 2.6.18-128.1.1.el5 and 2.6.18-164.2.1.el5 >>>>>>>>>> >>>>>>>>>> Any suggestions? >>>>>>>>>> Thank you very much in advance. >>>>>>>>>> >>>>>>>>>> Mirko Corosu >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> >>>>>>>>>> -------------------------------------------------------------------- >>>> -- >>>>>> -- >>>>>>>> -- >>>>>>>>>> Mirko Corosu >>>>>>>>>> Network and system administrator >>>>>>>>>> Computing Center >>>>>>>>>> Istituto Nazionale Fisica Nucleare >>>>>>>>>> Via Dodecaneso 33 >>>>>>>>>> 16146 Genova, Italy >>>>>>>>>> http://www.ge.infn.it >>>>>>>>>> -------------------------------------------------------------------- >>>> -- >>>>>> -- >>>>>>>> -- >>>>>>>>>> -------------------------------------------------------------------- >>>> -- >>>>>> -- >>>>>>>> --- >>>>>>>>>> --- >>>>>>>>>> Download Intel® Parallel Studio Eval >>>>>>>>>> Try the new software tools for yourself. Speed compiling, find bugs >>>>>>>>>> proactively, and fine-tune applications for parallel performance. >>>>>>>>>> See why Intel Parallel Studio got high marks during beta. >>>>>>>>>> http://p.sf.net/sfu/intel-sw-dev >>>>>>>>>> _______________________________________________ >>>>>>>>>> E1000-devel mailing list >>>>>>>>>> E10...@li... >>>>>>>>>> https://lists.sourceforge.net/lists/listinfo/e1000-devel >>>>>>>>>> To learn more about Intel® Ethernet, visit >>>>>>>>>> http://communities.intel.com/community/wired >>>>>>>> -- >>>>>>>> >>>>>>>> ---------------------------------------------------------------------- >>>> -- >>>>>> -- >>>>>>>> Mirko Corosu >>>>>>>> Istituto Nazionale Fisica Nucleare >>>>>>>> Via Dodecaneso 33 >>>>>>>> 16146 Genova >>>>>>>> www.ge.infn.it >>>>>>>> Phone +39 010 3536361 >>>>>>>> ---------------------------------------------------------------------- >>>> -- >>>>>> -- >>>>>>>> ---------------------------------------------------------------------- >>>> -- >>>>>> -- >>>>>>>> Se tutto sembra venirti incontro, probabilmente sei nella corsia >>>>>> sbagliata. >>>>>>>> ---------------------------------------------------------------------- >>>> -- >>>>>> -- >>>>>> >>>>>> -- >>>>>> >>>>>> ------------------------------------------------------------------------ >>>> -- >>>>>> Mirko Corosu >>>>>> Network and system administrator >>>>>> Computing Center >>>>>> Istituto Nazionale Fisica Nucleare >>>>>> Via Dodecaneso 33 >>>>>> 16146 Genova, Italy >>>>>> http://www.ge.infn.it >>>>>> ------------------------------------------------------------------------ >>>> -- >>>>>> >>>>>> ------------------------------------------------------------------------ >>>> -- >>>>>> Se tutto sembra venirti incontro, probabilmente sei nella corsia >>>> sbagliata. >>>>>> ------------------------------------------------------------------------ >>>> -- >>>> >>>> -- >>>> -------------------------------------------------------------------------- >>>> Mirko Corosu >>>> Network and system administrator >>>> Computing Center >>>> Istituto Nazionale Fisica Nucleare >>>> Via Dodecaneso 33 >>>> 16146 Genova, Italy >>>> http://www.ge.infn.it >>>> -------------------------------------------------------------------------- >>>> >>>> -------------------------------------------------------------------------- >>>> Se tutto sembra venirti incontro, probabilmente sei nella corsia sbagliata. >>>> -------------------------------------------------------------------------- >>>> >>>> >>>> >>>> --------------------------------------------------------------------------- >>>> --- >>>> Download Intel® Parallel Studio Eval >>>> Try the new software tools for yourself. Speed compiling, find bugs >>>> proactively, and fine-tune applications for parallel performance. >>>> See why Intel Parallel Studio got high marks during beta. >>>> http://p.sf.net/sfu/intel-sw-dev >>>> _______________________________________________ >>>> E1000-devel mailing list >>>> E10...@li... >>>> https://lists.sourceforge.net/lists/listinfo/e1000-devel >>>> To learn more about Intel® Ethernet, visit >>>> http://communities.intel.com/community/wired >> >> -- >> -------------------------------------------------------------------------- >> Mirko Corosu >> Network and system administrator >> Computing Center >> Istituto Nazionale Fisica Nucleare >> Via Dodecaneso 33 >> 16146 Genova, Italy >> http://www.ge.infn.it >> -------------------------------------------------------------------------- >> >> -------------------------------------------------------------------------- >> Se tutto sembra venirti incontro, probabilmente sei nella corsia sbagliata. >> -------------------------------------------------------------------------- >> >> >> ------------------------------------------------------------------------------ >> Download Intel® Parallel Studio Eval >> Try the new software tools for yourself. Speed compiling, find bugs >> proactively, and fine-tune applications for parallel performance. >> See why Intel Parallel Studio got high marks during beta. >> http://p.sf.net/sfu/intel-sw-dev >> _______________________________________________ >> E1000-devel mailing list >> E10...@li... >> https://lists.sourceforge.net/lists/listinfo/e1000-devel >> To learn more about Intel® Ethernet, visit http://communities.intel.com/community/wired > -- -------------------------------------------------------------------------- Mirko Corosu Network and system administrator Computing Center Istituto Nazionale Fisica Nucleare Via Dodecaneso 33 16146 Genova, Italy http://www.ge.infn.it -------------------------------------------------------------------------- |
