Menu
▾
▴
Re: [E1000-devel] Intel 2598EB 10-Gigabit AT dropped rx packet
|
From: Skidmore, D. C <don...@in...> - 2010-02-24 00:10:17
|
Hi Mirko, Your information was fine; I just wasn't asking detailed enough questions. :) I think I got off track focusing on the rx_missed_errors when it sounds like you're more concerned on why you're not getting better throughput. One thing I did notice was that the Receive side has 8 rx queues while the Transmit has 16 rx queues. You mentioned these were identical machines could Hyper-Threading be off on the Receiving machine? Also I'm still a little confused about how your test is set up. If you have four netperf's running on, as you called it, your Tx machine and your Rx is only running netserver then I would only expect heavy traffic from Tx -> Rx. Or am I missing some thing here? Thanks, -Don >-----Original Message----- >From: Mirko Corosu [mailto:mir...@ge...] >Sent: Tuesday, February 23, 2010 3:30 AM >To: Skidmore, Donald C >Cc: E10...@li... >Subject: Re: [E1000-devel] Intel 2598EB 10-Gigabit AT dropped rx packet > >Hi Donald, > >Ok, sorry for the lack of informations: I'm not used to interact with >developers. :-) > > >> How were you using netperf in your tests? What tests (I assume >[UDP|TCP]_STREAM) and how many instances of netperf were running each side? >> > >I'm running four instances of UDP_STREAM or TCP_STREAM tests on the tx >side, with different packet sizes (from 1000 to 9000 byte) and a single >netserver on the rx side. > >> >> From the spec I got off the web on the R710 I can see they have either (2 >PCIe x8 and 2 PCIe x4) or (1 PCIe x16 and 2 PCIe x4 ). I was just >concerned you might be plugged into one of the x4 slots. > >I have 2 PCIe x8 on the so called "Riser 2" and 2 PCIe x4 on the "Riser >1". On each server the Intel cards are plugged into the Riser 2 slots. > >> >>>> - Did you look at the dmesg for anything relative? >>> I see no error or warning messages on both side. >> >> It's not just error or warning messages I was interested in, although >they would be interesting too. :) We also printk a message that displays >the type of PCIe link we have. This would answer the question I posed >above about what slot the cards were in. >> > >On rx side: > >[root@grids2 ~]# dmesg |grep ixgbe >ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version >2.0.44.14-NAPI >ixgbe: 0000:07:00.0: ixgbe_init_interrupt_scheme: Multiqueue Enabled: Rx >Queue count = 8, Tx Queue count = 1 >ixgbe: eth0: ixgbe_probe: (PCI Express:2.5Gb/s:Width x8) 00:1b:21:4b:c8:bf >ixgbe: eth0: ixgbe_probe: MAC: 1, PHY: 2, PBA No: d79893-017 >ixgbe: eth0: ixgbe_probe: Internal LRO is enabled >ixgbe: eth0: ixgbe_probe: Intel(R) 10 Gigabit Network Connection >ixgbe: eth0: ixgbe_change_mtu: changing MTU from 1500 to 9000 >ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Up 10 Gbps, Flow Control: >RX/TX >ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Up 10 Gbps, Flow Control: >None >ixgbe: eth0: ixgbe_remove: complete >ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version >2.0.44.14-NAPI >ixgbe: 0000:07:00.0: ixgbe_init_interrupt_scheme: Multiqueue Enabled: Rx >Queue count = 8, Tx Queue count = 1 >ixgbe: eth0: ixgbe_probe: (PCI Express:2.5Gb/s:Width x8) 00:1b:21:4b:c8:bf >ixgbe: eth0: ixgbe_probe: MAC: 1, PHY: 2, PBA No: d79893-017 >ixgbe: eth0: ixgbe_probe: Internal LRO is enabled >ixgbe: eth0: ixgbe_probe: Intel(R) 10 Gigabit Network Connection >ixgbe: eth0: ixgbe_change_mtu: changing MTU from 1500 to 9000 >ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Up 10 Gbps, Flow Control: >RX/TX >ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Up 10 Gbps, Flow Control: >None >ixgbe: eth0: ixgbe_remove: complete >ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version >2.0.44.14-NAPI >ixgbe: 0000:07:00.0: ixgbe_init_interrupt_scheme: Multiqueue Enabled: Rx >Queue count = 8, Tx Queue count = 1 >ixgbe: eth0: ixgbe_probe: (PCI Express:2.5Gb/s:Width x8) 00:1b:21:4b:c8:bf >ixgbe: eth0: ixgbe_probe: MAC: 1, PHY: 2, PBA No: d79893-017 >ixgbe: eth0: ixgbe_probe: Internal LRO is enabled >ixgbe: eth0: ixgbe_probe: Intel(R) 10 Gigabit Network Connection >ixgbe: eth0: ixgbe_change_mtu: changing MTU from 1500 to 9000 >ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Up 10 Gbps, Flow Control: >RX/TX >ixgbe: eth0: ixgbe_remove: complete >ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version >2.0.44.14-NAPI >ixgbe: 0000:07:00.0: ixgbe_init_interrupt_scheme: Multiqueue Enabled: Rx >Queue count = 8, Tx Queue count = 1 >ixgbe: eth0: ixgbe_probe: (PCI Express:2.5Gb/s:Width x8) 00:1b:21:4b:c8:bf >ixgbe: eth0: ixgbe_probe: MAC: 1, PHY: 2, PBA No: d79893-017 >ixgbe: eth0: ixgbe_probe: Internal LRO is enabled >ixgbe: eth0: ixgbe_probe: Intel(R) 10 Gigabit Network Connection >ixgbe: eth0: ixgbe_change_mtu: changing MTU from 1500 to 9000 >ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Up 10 Gbps, Flow Control: >RX/TX >ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Up 10 Gbps, Flow Control: >None >ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Up 10 Gbps, Flow Control: >RX/TX >ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Down >ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Up 10 Gbps, Flow Control: >RX/TX >ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Down >ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Up 10 Gbps, Flow Control: >RX/TX >ixgbe: eth0: ixgbe_remove: complete >ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version >2.0.44.14-NAPI >ixgbe: 0000:07:00.0: ixgbe_init_interrupt_scheme: Multiqueue Enabled: Rx >Queue count = 8, Tx Queue count = 1 >ixgbe: eth0: ixgbe_probe: (PCI Express:2.5Gb/s:Width x8) 00:1b:21:4b:c8:bf >ixgbe: eth0: ixgbe_probe: MAC: 1, PHY: 2, PBA No: d79893-017 >ixgbe: eth0: ixgbe_probe: Internal LRO is enabled >ixgbe: eth0: ixgbe_probe: Intel(R) 10 Gigabit Network Connection >ixgbe: eth0: ixgbe_change_mtu: changing MTU from 1500 to 9000 >ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Up 10 Gbps, Flow Control: >RX/TX >ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Down >ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Up 10 Gbps, Flow Control: >RX/TX >ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Down >ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Up 10 Gbps, Flow Control: >RX/TX >ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Down >ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Up 10 Gbps, Flow Control: >RX/TX >ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Down >ixgbe: eth0: ixgbe_remove: complete >ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version >2.0.44.14-NAPI >ixgbe: 0000:07:00.0: ixgbe_init_interrupt_scheme: Multiqueue Enabled: Rx >Queue count = 8, Tx Queue count = 1 >ixgbe: eth0: ixgbe_probe: (PCI Express:2.5Gb/s:Width x8) 00:1b:21:4b:c8:bf >ixgbe: eth0: ixgbe_probe: MAC: 1, PHY: 2, PBA No: d79893-017 >ixgbe: eth0: ixgbe_probe: Internal LRO is enabled >ixgbe: eth0: ixgbe_probe: Intel(R) 10 Gigabit Network Connection >ixgbe: eth0: ixgbe_change_mtu: changing MTU from 1500 to 9000 >ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Up 10 Gbps, Flow Control: >RX/TX > > > >On tx side: > >[root@client20 ~]# dmesg |grep ixgbe >ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version >2.0.44.14-NAPI >ixgbe: 0000:07:00.0: ixgbe_init_interrupt_scheme: Multiqueue Enabled: Rx >Queue count = 16, Tx Queue count = 1 >ixgbe: eth0: ixgbe_probe: No DCA provider found. Please start ioatdma >for DCA functionality. >ixgbe: eth0: ixgbe_probe: (PCI Express:2.5Gb/s:Width x8) 00:1b:21:4b:c8:e0 >ixgbe: eth0: ixgbe_probe: MAC: 1, PHY: 2, PBA No: d79893-017 >ixgbe: eth0: ixgbe_probe: Internal LRO is enabled >ixgbe: eth0: ixgbe_probe: Intel(R) 10 Gigabit Network Connection >ixgbe: eth0: ixgbe_change_mtu: changing MTU from 1500 to 9000 >ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Up 10 Gbps, Flow Control: >None >ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Up 10 Gbps, Flow Control: >RX/TX >ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Down >ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Up 10 Gbps, Flow Control: >RX/TX >ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Down >ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Up 10 Gbps, Flow Control: >RX/TX >ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Down >ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Up 10 Gbps, Flow Control: >RX/TX >ixgbe: eth0: ixgbe_remove: complete >ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version >2.0.44.14-NAPI >ixgbe: 0000:07:00.0: ixgbe_init_interrupt_scheme: Multiqueue Enabled: Rx >Queue count = 16, Tx Queue count = 1 >ixgbe: eth0: ixgbe_probe: No DCA provider found. Please start ioatdma >for DCA functionality. >ixgbe: eth0: ixgbe_probe: (PCI Express:2.5Gb/s:Width x8) 00:1b:21:4b:c8:e0 >ixgbe: eth0: ixgbe_probe: MAC: 1, PHY: 2, PBA No: d79893-017 >ixgbe: eth0: ixgbe_probe: Internal LRO is enabled >ixgbe: eth0: ixgbe_probe: Intel(R) 10 Gigabit Network Connection >ixgbe: eth0: ixgbe_change_mtu: changing MTU from 1500 to 9000 >ixgbe: eth0: ixgbe_remove: complete >ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version >2.0.44.14-NAPI >ixgbe: 0000:07:00.0: ixgbe_init_interrupt_scheme: Multiqueue Enabled: Rx >Queue count = 16, Tx Queue count = 1 >ixgbe: eth0: ixgbe_probe: No DCA provider found. Please start ioatdma >for DCA functionality. >ixgbe: eth0: ixgbe_probe: (PCI Express:2.5Gb/s:Width x8) 00:1b:21:4b:c8:e0 >ixgbe: eth0: ixgbe_probe: MAC: 1, PHY: 2, PBA No: d79893-017 >ixgbe: eth0: ixgbe_probe: Internal LRO is enabled >ixgbe: eth0: ixgbe_probe: Intel(R) 10 Gigabit Network Connection >ixgbe: eth0: ixgbe_change_mtu: changing MTU from 1500 to 9000 >ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Up 10 Gbps, Flow Control: >RX/TX >ixgbe: eth0: ixgbe_remove: complete >ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version >2.0.44.14-NAPI >ixgbe: 0000:07:00.0: ixgbe_init_interrupt_scheme: Multiqueue Enabled: Rx >Queue count = 16, Tx Queue count = 1 >ixgbe: eth0: ixgbe_probe: No DCA provider found. Please start ioatdma >for DCA functionality. >ixgbe: eth0: ixgbe_probe: (PCI Express:2.5Gb/s:Width x8) 00:1b:21:4b:c8:e0 >ixgbe: eth0: ixgbe_probe: MAC: 1, PHY: 2, PBA No: d79893-017 >ixgbe: eth0: ixgbe_probe: Internal LRO is enabled >ixgbe: eth0: ixgbe_probe: Intel(R) 10 Gigabit Network Connection >ixgbe: eth0: ixgbe_change_mtu: changing MTU from 1500 to 9000 >ixgbe: eth0: ixgbe_watchdog_task: NIC Link is Up 10 Gbps, Flow Control: >RX/TX > > > >> >> All that said we would expect to see rx_missed_errors with flow control >disabled. Since nothing would be stopping packets from overrunning the >buffer. Your test seems to be agreeing with this as the number of >rx_missed_errors goes down with TCP (like you said) most likely due to TCP >sliding window. >> >> You don't see this issue with flow control enabled, right? > >Right, but the throughput remains about 5.6 Gb/s and a can't figure out >where the bottleneck could be.... > >Thanks a lot > >Mirko > >> >>> -----Original Message----- >>> From: Mirko Corosu [mailto:mir...@ge...] >>> Sent: Saturday, February 20, 2010 10:50 AM >>> To: Skidmore, Donald C >>> Cc: E10...@li... >>> Subject: Re: [E1000-devel] Intel 2598EB 10-Gigabit AT dropped rx packet >>> >>> Hi Donald, >>> >>> Thank you for your reply. >>> >>>> I have a couple additional questions >>>> - What were you running to do this test? (i.e. netperf, pktgen?) >>> netperf-2.4.2-1.el5.rf >>> >>>> - What throughput were you seeing? >>> On tx side i see almost 10Gb/s, on rx side I see about 5.6 Gb/s >>> >>>> - Are both ports plugged into x8 PCIe slots >>> Yes, both servers are Dell R710 with the same hardware configuration >>> >>>> - Did you look at the dmesg for anything relative? >>> I see no error or warning messages on both side. >>> >>>> - Do you see this issue only with UDP? >>> If I send a TCP stream I see on both side a throughput of about 5.54 >>> Gb/s and less rx_missed_errors on rx side: >>> >>> NIC statistics: >>> rx_packets: 1768101 >>> tx_packets: 235384 >>> rx_bytes: 15937257320 >>> tx_bytes: 15704254 >>> lsc_int: 2 >>> tx_busy: 0 >>> non_eop_descs: 7072222 >>> rx_errors: 0 >>> tx_errors: 0 >>> rx_dropped: 0 >>> tx_dropped: 0 >>> multicast: 1 >>> broadcast: 0 >>> rx_no_buffer_count: 0 >>> collisions: 0 >>> rx_over_errors: 0 >>> rx_crc_errors: 0 >>> rx_frame_errors: 0 >>> rx_fifo_errors: 0 >>> rx_missed_errors: 1539 >>> tx_aborted_errors: 0 >>> tx_carrier_errors: 0 >>> tx_fifo_errors: 0 >>> tx_heartbeat_errors: 0 >>> tx_timeout_count: 0 >>> tx_restart_queue: 0 >>> rx_long_length_errors: 0 >>> rx_short_length_errors: 0 >>> tx_tcp4_seg_ctxt: 0 >>> tx_tcp6_seg_ctxt: 0 >>> tx_flow_control_xon: 0 >>> rx_flow_control_xon: 0 >>> tx_flow_control_xoff: 0 >>> rx_flow_control_xoff: 0 >>> rx_csum_offload_good: 1768098 >>> rx_csum_offload_errors: 0 >>> tx_csum_offload_ctxt: 11 >>> low_latency_interrupt: 0 >>> alloc_rx_page_failed: 0 >>> alloc_rx_buff_failed: 0 >>> lro_aggregated: 1760417 >>> lro_flushed: 375277 >>> lro_recycled: 1012791 >>> rx_no_dma_resources: 0 >>> hw_rsc_count: 0 >>> rx_flm: 0 >>> fdir_match: 0 >>> fdir_miss: 0 >>> tx_queue_0_packets: 235384 >>> tx_queue_0_bytes: 15704254 >>> rx_queue_0_packets: 2 >>> rx_queue_0_bytes: 120 >>> rx_queue_1_packets: 8 >>> rx_queue_1_bytes: 786 >>> rx_queue_2_packets: 301752 >>> rx_queue_2_bytes: 2719959436 >>> rx_queue_3_packets: 562919 >>> rx_queue_3_bytes: 5074118826 >>> rx_queue_4_packets: 7 >>> rx_queue_4_bytes: 726 >>> rx_queue_5_packets: 15 >>> rx_queue_5_bytes: 1526 >>> rx_queue_6_packets: 313901 >>> rx_queue_6_bytes: 2829476778 >>> rx_queue_7_packets: 589497 >>> rx_queue_7_bytes: 5313699122 >>> >>> I guess it's because of TCP flow control. >>> >>> I remember that the same behavior was triggered by enabling flow control >>> on both cards (ethtool -A eth0 rx on tx on autoconf on) but I have to >>> wait till Monday to perform another run to test that configuration. >>> >>> Thanks, >>> >>> Mirko >>> >>>> What I'm wondering is if this NIC was receiving faster than it could >>> process them. >>>> Thanks, >>>> -Don Skidmore <Don...@in...> >>>> >>>> >>>>> -----Original Message----- >>>>> From: Mirko Corosu [mailto:mir...@ge...] >>>>> Sent: Friday, February 19, 2010 5:30 AM >>>>> To: E10...@li... >>>>> Subject: [E1000-devel] Intel 2598EB 10-Gigabit AT dropped rx packet >>>>> >>>>> Dear all, >>>>> >>>>> I'm new to this list, if this isn't the right place to post this >issue, >>>>> please let me know. >>>>> >>>>> I am testing two Intel 82598EB 10-Gigabit AT network card mounted on >two >>>>> Dell R710 server with Red Hat Enterprise Linux 5.4 installed. The two >>>>> NICs are directly connected each other (no switch in between). I am >>>>> experiencing a problem on the receive side of the connection: I'm >>> sending >>>>> an UDP stream after having disabled flow control on each card, and >about >>>>> the 60% of the transmitted packets are dropped with a large number of >>>>> rx_missed_errors. This is what ifconfig and ethtool -S show: >>>>> >>>>> [root@grids2]# ethtool -S eth0 >>>>> NIC statistics: >>>>> rx_packets: 1761794 >>>>> tx_packets: 32 >>>>> rx_bytes: 14748667944 >>>>> tx_bytes: 5746 >>>>> lsc_int: 2 >>>>> tx_busy: 0 >>>>> non_eop_descs: 6510828 >>>>> rx_errors: 0 >>>>> tx_errors: 0 >>>>> rx_dropped: 0 >>>>> tx_dropped: 0 >>>>> multicast: 2 >>>>> broadcast: 2 >>>>> rx_no_buffer_count: 0 >>>>> collisions: 0 >>>>> rx_over_errors: 0 >>>>> rx_crc_errors: 0 >>>>> rx_frame_errors: 0 >>>>> rx_fifo_errors: 0 >>>>> rx_missed_errors: 1148382 >>>>> tx_aborted_errors: 0 >>>>> tx_carrier_errors: 0 >>>>> tx_fifo_errors: 0 >>>>> tx_heartbeat_errors: 0 >>>>> tx_timeout_count: 0 >>>>> tx_restart_queue: 0 >>>>> rx_long_length_errors: 0 >>>>> rx_short_length_errors: 0 >>>>> tx_tcp4_seg_ctxt: 0 >>>>> tx_tcp6_seg_ctxt: 0 >>>>> tx_flow_control_xon: 0 >>>>> rx_flow_control_xon: 0 >>>>> tx_flow_control_xoff: 0 >>>>> rx_flow_control_xoff: 0 >>>>> rx_csum_offload_good: 28 >>>>> rx_csum_offload_errors: 0 >>>>> tx_csum_offload_ctxt: 11 >>>>> low_latency_interrupt: 0 >>>>> alloc_rx_page_failed: 0 >>>>> alloc_rx_buff_failed: 0 >>>>> lro_aggregated: 8 >>>>> lro_flushed: 8 >>>>> lro_recycled: 0 >>>>> rx_no_dma_resources: 0 >>>>> hw_rsc_count: 0 >>>>> rx_flm: 0 >>>>> fdir_match: 0 >>>>> fdir_miss: 0 >>>>> tx_queue_0_packets: 32 >>>>> tx_queue_0_bytes: 5746 >>>>> rx_queue_0_packets: 2 >>>>> rx_queue_0_bytes: 120 >>>>> rx_queue_1_packets: 2 >>>>> rx_queue_1_bytes: 120 >>>>> rx_queue_2_packets: 1761762 >>>>> rx_queue_2_bytes: 14748664800 >>>>> rx_queue_3_packets: 0 >>>>> rx_queue_3_bytes: 0 >>>>> rx_queue_4_packets: 0 >>>>> rx_queue_4_bytes: 0 >>>>> rx_queue_5_packets: 14 >>>>> rx_queue_5_bytes: 1452 >>>>> rx_queue_6_packets: 7 >>>>> rx_queue_6_bytes: 726 >>>>> rx_queue_7_packets: 7 >>>>> rx_queue_7_bytes: 726 >>>>> >>>>> [root@grids2]# ifconfig eth0 >>>>> >>>>> eth0 Link encap:Ethernet HWaddr 00:1B:21:4B:C8:BF >>>>> inet addr:10.100.100.2 Bcast:10.100.100.255 >>> Mask:255.255.255.0 >>>>> inet6 addr: fe80::21b:21ff:fe4b:c8bf/64 Scope:Link >>>>> UP BROADCAST RUNNING MULTICAST MTU:9000 Metric:1 >>>>> RX packets:1761795 errors:0 dropped:1148382 overruns:0 >frame:0 >>>>> TX packets:39 errors:0 dropped:0 overruns:0 carrier:0 >>>>> collisions:0 txqueuelen:1000 >>>>> RX bytes:14748668004 (13.7 GiB) TX bytes:9332 (9.1 KiB) >>>>> >>>>> Each interface has the MTU set to 9000, the driver is the one provided >>>>> by Dell support (ixgbe 2.0.44.14-NAPI) with Rx multiqueue enabled and >>>>> InterruptThrottleRate set to 8000 (but I tried 16000 and 32000 too). >>>>> Also I tried to set FdirPballoc parameter to 0, 1 and 2 with no >success. >>>>> Single CPU load is no more than 10% with 8000 interrupt/s: >>>>> >>>>> [root@grids2 ~]# mpstat -P ALL 2 >>>>> >>>>> 05:16:43 PM CPU %user %nice %sys %iowait %irq %soft >%steal >>>>> %idle intr/s >>>>> 05:16:45 PM all 0.00 0.00 0.06 0.00 0.06 1.19 >0.00 >>>>> 98.69 9018.00 >>>>> 05:16:45 PM 0 0.00 0.00 0.00 0.00 0.00 0.00 >0.00 >>>>> 100.00 1002.50 >>>>> 05:16:45 PM 1 0.00 0.00 0.00 0.00 0.00 0.00 >0.00 >>>>> 100.00 1.50 >>>>> 05:16:45 PM 2 0.00 0.00 0.50 0.00 0.50 9.45 >0.00 >>>>> 89.55 8007.00 >>>>> 05:16:45 PM 3 0.00 0.00 0.00 0.00 0.00 0.00 >0.00 >>>>> 100.00 1.00 >>>>> 05:16:45 PM 4 0.00 0.00 0.00 0.00 0.00 0.00 >0.00 >>>>> 100.00 1.00 >>>>> 05:16:45 PM 5 0.00 0.00 0.00 0.00 0.00 0.00 >0.00 >>>>> 100.00 3.00 >>>>> 05:16:45 PM 6 0.00 0.00 0.00 0.00 0.00 0.00 >0.00 >>>>> 100.00 1.00 >>>>> 05:16:45 PM 7 0.00 0.00 0.00 0.00 0.00 0.00 >0.00 >>>>> 100.00 1.00 >>>>> >>>>> I tried to configure smp affinity to assign the IRQ of each >>>>> rx-queue to a different CPU, with no effect on the amount of dropped >>>>> packets. I also verified that if I connect the NIC on a 10GE switch >and >>>>> send more than one stream from different servers, the card uses >>>>> different rx-queues but the problem still exists. >>>>> >>>>> I modified the /etc/sysctl.conf file parameters adding: >>>>> >>>>> net.core.wmem_max = 16777216 >>>>> net.core.wmem_default = 8388608 >>>>> net.core.rmem_max = 16777216 >>>>> net.core.rmem_default = 8388608 >>>>> net.ipv4.tcp_rmem = 4096 262144 16777216 >>>>> net.ipv4.tcp_wmem = 4096 262144 16777216 >>>>> net.ipv4.tcp_mem = 4096 8388608 16777216 >>>>> net.core.optmem_max = 524288 >>>>> net.core.netdev_max_backlog = 200000 >>>>> >>>>> >>>>> >>>>> >>>>> Some other details: >>>>> >>>>> - CPU: dual quad core Intel L5520 @ 2.27GHz (Hyper-Threading >disabled) >>>>> - RAM: 24 GB (12 X 2GB) >>>>> - Slot PCI: PCIe gen.2 x8 >>>>> - Kernel linux: 2.6.18-128.1.1.el5 and 2.6.18-164.2.1.el5 >>>>> >>>>> Any suggestions? >>>>> Thank you very much in advance. >>>>> >>>>> Mirko Corosu >>>>> >>>>> -- >>>>> >>>>> ---------------------------------------------------------------------- >-- >>> -- >>>>> Mirko Corosu >>>>> Network and system administrator >>>>> Computing Center >>>>> Istituto Nazionale Fisica Nucleare >>>>> Via Dodecaneso 33 >>>>> 16146 Genova, Italy >>>>> http://www.ge.infn.it >>>>> ---------------------------------------------------------------------- >-- >>> -- >>>>> >>>>> >>>>> ---------------------------------------------------------------------- >-- >>> --- >>>>> --- >>>>> Download Intel® Parallel Studio Eval >>>>> Try the new software tools for yourself. Speed compiling, find bugs >>>>> proactively, and fine-tune applications for parallel performance. >>>>> See why Intel Parallel Studio got high marks during beta. >>>>> http://p.sf.net/sfu/intel-sw-dev >>>>> _______________________________________________ >>>>> E1000-devel mailing list >>>>> E10...@li... >>>>> https://lists.sourceforge.net/lists/listinfo/e1000-devel >>>>> To learn more about Intel® Ethernet, visit >>>>> http://communities.intel.com/community/wired >>> >>> -- >>> >>> ------------------------------------------------------------------------ >-- >>> Mirko Corosu >>> Istituto Nazionale Fisica Nucleare >>> Via Dodecaneso 33 >>> 16146 Genova >>> www.ge.infn.it >>> Phone +39 010 3536361 >>> ------------------------------------------------------------------------ >-- >>> >>> ------------------------------------------------------------------------ >-- >>> Se tutto sembra venirti incontro, probabilmente sei nella corsia >sbagliata. >>> ------------------------------------------------------------------------ >-- >> > > >-- > >-------------------------------------------------------------------------- >Mirko Corosu >Network and system administrator >Computing Center >Istituto Nazionale Fisica Nucleare >Via Dodecaneso 33 >16146 Genova, Italy >http://www.ge.infn.it >-------------------------------------------------------------------------- > > > >-------------------------------------------------------------------------- >Se tutto sembra venirti incontro, probabilmente sei nella corsia sbagliata. >-------------------------------------------------------------------------- |
