Menu
▾
▴
Re: [E1000-devel] Problems with Dell R810.
|
From: <and...@ag...> - 2011-09-13 14:17:18
|
Hi Donald,
I removed the second 10 Gig NIC and switched off the second pair of built-in 1 Gig via the BIOS so as to reduce the bus loading, as a result I now have 4 NIC active. Unfortunately, this change didn't seem to make much difference to performance.
Next I reduced the number of active cores (processors) in the system, again using the BIOS settings, and switched Hyper Threading off. Here's the results of 10,000,000 packets 64 bytes per packet, listed is the highest throughput before packets were dropped.
1 Queue num proc Queues
4 procs 110Mbits/s 125Mbits/s
8 procs 110Mbits/s 215Mbits/s
16 procs 110 Mbits/s 180Mbits/s
When running with "num proc" queues I simply used the default driver values with i.e. "insmod ixgbe.ko", so that the number of queues matched the number of cores. I used the ixgbe-3.4.24 driver.
I did notice that on 16 cores when the packets started dropping, the CPU usage of the processes labeled ksoftirqd/proc_num i.e. "ksoftirqd/0, ksoftirqd/1 and so on through 15" when through the roof so that before packets were dropped those processes barely registered in 'top' but as soon as packets were dropped these processes dominated the output of top, almost like they were thrashing.
In a simple experiment to prove this, I noted that in the case of 16 processors at 175Mbits/s these processes were under 1% CPU (mostly 0.0%) use but at, say 190Mbits/s these processing dominated the output of top, most at > 90% with some in the mid 50%; it was almost like they were thrashing.
Again I am seeing the kind of throughput I was before, I guess this is much less than you'd expect?
Hope this helps...
Andrew
-----Original Message-----
From: and...@ag... [mailto:and...@ag...]
Sent: Thursday, September 08, 2011 3:47 PM
To: don...@in...; de...@nt...
Cc: e10...@li...
Subject: Re: [E1000-devel] Problems with Dell R810.
Hi Donald,
I ran a small program that uses libpcap to capture promiscuously from one queue on one core, it uses a 1 second interval sigalarm to monitor capture progress and will exit 5 seconds after no packets have been seen (basically a trigger once the source has stopped sending). On exit it prints out the number of packets observed in the run.
I used the latest driver ixgbe-3.4.24, this was without PF_RING. The program simply grabbed the packets and counted bytes, it barely managed 100Megabits/s before it started dropping packets; the following is from a run at 111Megabits/s.
I sent 10,000,000 packets form an external test source, packet size was 64 bytes.
My app saw 9,562,113 packets
Some of The dropped packets are seen with ethtool -S in
rx_missed_errors: 48331
..
rx_pkts_nic: 9951669
I used the script in "../scripts/set_irq_affinity.sh" to pin the interrupts to the processor.
Here's a screen scrape from 'top' for the process and the interrupt.
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3 root 20 0 0 0 0 S 35.2 0.0 13:41.11 ksoftirqd/0
36037 root 20 0 8616 688 588 R 25.1 0.0 0:05.03 count_pcap
Here's the log file (ps please do read on, as there's more results after the log).
Sep 8 11:01:57 kernel: [260638.894886] Intel(R) 10 Gigabit PCI Express Network Driver - version 3.4.24-NAPI Sep 8 11:01:57 kernel: [260638.894893] Copyright (c) 1999-2011 Intel Corporation.
Sep 8 11:01:57 kernel: [260638.894966] ixgbe 0000:0e:00.0: PCI INT A -> GSI 38 (level, low) -> IRQ 38 Sep 8 11:01:57 kernel: [260639.020109] ixgbe: Multiple Queue Support Disabled Sep 8 11:01:57 kernel: [260639.020116] ixgbe: 0000:0e:00.0: ixgbe_check_options: RSS is not supported while multiple queues are disabled. Disabling RSS.
Sep 8 11:01:57 kernel: [260639.020123] ixgbe: 0000:0e:00.0: ixgbe_check_options: Flow Director is not supported while multiple queues are disabled. Disabling Flow Director Sep 8 11:01:57 kernel: [260639.020130] ixgbe: 0000:0e:00.0: ixgbe_check_options: FCoE Offload feature enabled Sep 8 11:01:57 kernel: [260639.044593] ixgbe 0000:0e:00.0: (PCI Express:5.0GT/s:Width x8) 00:1b:21:b5:c0:b0 Sep 8 11:01:57 kernel: [260639.044906] ixgbe 0000:0e:00.0: eth4: MAC: 2, PHY: 15, SFP+: 5, PBA No: G18786-001 Sep 8 11:01:57 kernel: [260639.044913] ixgbe 0000:0e:00.0: eth4: Enabled Features: RxQ: 1 TxQ: 1 RSC Sep 8 11:01:57 kernel: [260639.052005] ixgbe 0000:0e:00.0: eth4: Intel(R) 10 Gigabit Network Connection Sep 8 11:01:57 kernel: [260639.052040] ixgbe 0000:0e:00.1: PCI INT B -> GSI 45 (level, low) -> IRQ 45 Sep 8 11:01:57 NetworkManager[2060]: <info> (eth4): carrier is OFF Sep 8 11:01:57 NetworkManager[2060]: <info> (eth4): new Ethernet device (driver: 'ixgbe' ifindex: 50) Sep 8 11:01:57 NetworkManager[2060]: <info> (eth4): exported as /org/freedesktop/NetworkManager/Devices/48
Sep 8 11:01:57 NetworkManager[2060]: <info> (eth4): now managed Sep 8 11:01:57 NetworkManager[2060]: <info> (eth4): device state change: 1 -> 2 (reason 2) Sep 8 11:01:57 NetworkManager[2060]: <info> (eth4): bringing up device.
Sep 8 11:01:58 NetworkManager[2060]: <info> (eth4): preparing device.
Sep 8 11:01:58 NetworkManager[2060]: <info> (eth4): deactivating device (reason: 2).
Sep 8 11:01:58 kernel: [260639.196124] ADDRCONF(NETDEV_UP): eth4: link is not ready Sep 8 11:01:58 kernel: [260639.235301] ixgbe 0000:0e:00.0: eth4: detected SFP+: 5 Sep 8 11:01:58 kernel: [260640.130940] ixgbe: Multiple Queue Support Disabled Sep 8 11:01:58 kernel: [260640.130948] ixgbe: 0000:0e:00.1: ixgbe_check_options: RSS is not supported while multiple queues are disabled. Disabling RSS.
Sep 8 11:01:58 kernel: [260640.130956] ixgbe: 0000:0e:00.1: ixgbe_check_options: Flow Director is not supported while multiple queues are disabled. Disabling Flow Director Sep 8 11:01:58 kernel: [260640.130963] ixgbe: 0000:0e:00.1: ixgbe_check_options: FCoE Offload feature enabled Sep 8 11:01:58 kernel: [260640.155466] ixgbe 0000:0e:00.1: (PCI Express:5.0GT/s:Width x8) 00:1b:21:b5:c0:b1 Sep 8 11:01:58 kernel: [260640.155777] ixgbe 0000:0e:00.1: eth5: MAC: 2, PHY: 1, PBA No: G18786-001 Sep 8 11:01:58 kernel: [260640.155785] ixgbe 0000:0e:00.1: eth5: Enabled Features: RxQ: 1 TxQ: 1 RSC Sep 8 11:01:59 kernel: [260640.162810] ixgbe 0000:0e:00.1: eth5: Intel(R) 10 Gigabit Network Connection Sep 8 11:01:59 kernel: [260640.162852] ixgbe 0000:0f:00.0: PCI INT A -> GSI 40 (level, low) -> IRQ 40 Sep 8 11:01:59 NetworkManager[2060]: <info> (eth5): carrier is OFF Sep 8 11:01:59 NetworkManager[2060]: <info> (eth5): new Ethernet device (driver: 'ixgbe' ifindex: 51) Sep 8 11:01:59 NetworkManager[2060]: <info> (eth5): exported as /org/freedesktop/NetworkManager/Devices/49
Sep 8 11:01:59 NetworkManager[2060]: <info> (eth5): now managed Sep 8 11:01:59 NetworkManager[2060]: <info> (eth5): device state change: 1 -> 2 (reason 2) Sep 8 11:01:59 NetworkManager[2060]: <info> (eth5): bringing up device.
Sep 8 11:01:59 kernel: [260640.286831] ixgbe: Multiple Queue Support Disabled Sep 8 11:01:59 kernel: [260640.286837] ixgbe: 0000:0f:00.0: ixgbe_check_options: RSS is not supported while multiple queues are disabled. Disabling RSS.
Sep 8 11:01:59 kernel: [260640.286843] ixgbe: 0000:0f:00.0: ixgbe_check_options: Flow Director is not supported while multiple queues are disabled. Disabling Flow Director Sep 8 11:01:59 kernel: [260640.286850] ixgbe: 0000:0f:00.0: ixgbe_check_options: FCoE Offload feature enabled Sep 8 11:01:59 kernel: [260640.286893] ixgbe 0000:0f:00.0: (unregistered net_device): FCoE offload feature is not available. Disabling FCoE offload feature Sep 8 11:01:59 NetworkManager[2060]: <info> (eth5): preparing device.
Sep 8 11:01:59 NetworkManager[2060]: <info> (eth5): deactivating device (reason: 2).
Sep 8 11:01:59 kernel: [260640.289025] ADDRCONF(NETDEV_UP): eth5: link is not ready Sep 8 11:01:59 kernel: [260640.311585] ixgbe 0000:0f:00.0: (PCI Express:5.0GT/s:Width x8) 00:e0:ed:ff:60:04 Sep 8 11:01:59 kernel: [260640.311678] ixgbe 0000:0f:00.0: eth6: MAC: 2, PHY: 12, SFP+: 5, PBA No: FFFFFF-0FF Sep 8 11:01:59 kernel: [260640.311689] ixgbe 0000:0f:00.0: eth6: Enabled Features: RxQ: 1 TxQ: 1 RSC Sep 8 11:01:59 kernel: [260640.311710] ixgbe 0000:0f:00.0: eth6: Intel(R) 10 Gigabit Network Connection Sep 8 11:01:59 kernel: [260640.311752] ixgbe 0000:0f:00.1: PCI INT B -> GSI 50 (level, low) -> IRQ 50 Sep 8 11:01:59 NetworkManager[2060]: <info> (eth6): carrier is OFF Sep 8 11:01:59 NetworkManager[2060]: <info> (eth6): new Ethernet device (driver: 'ixgbe' ifindex: 52) Sep 8 11:01:59 NetworkManager[2060]: <info> (eth6): exported as /org/freedesktop/NetworkManager/Devices/50
Sep 8 11:01:59 NetworkManager[2060]: <info> (eth6): now managed Sep 8 11:01:59 NetworkManager[2060]: <info> (eth6): device state change: 1 -> 2 (reason 2) Sep 8 11:01:59 NetworkManager[2060]: <info> (eth6): bringing up device.
Sep 8 11:01:59 NetworkManager[2060]: <info> (eth6): preparing device.
Sep 8 11:01:59 NetworkManager[2060]: <info> (eth6): deactivating device (reason: 2).
Sep 8 11:01:59 kernel: [260640.451779] ADDRCONF(NETDEV_UP): eth6: link is not ready Sep 8 11:01:59 kernel: [260640.495549] ixgbe 0000:0f:00.0: eth6: detected SFP+: 5 Sep 8 11:01:59 kernel: [260640.640869] ixgbe 0000:0f:00.0: eth6: NIC Link is Up 10 Gbps, Flow Control: RX/TX Sep 8 11:01:59 NetworkManager[2060]: <info> (eth6): carrier now ON (device state 2) Sep 8 11:01:59 NetworkManager[2060]: <info> (eth6): device state change: 2 -> 3 (reason 40) Sep 8 11:01:59 kernel: [260640.648957] ADDRCONF(NETDEV_CHANGE): eth6: link becomes ready Sep 8 11:01:59 NetworkManager[2060]: <info> Activation (eth6) starting connection 'System eth6'
Sep 8 11:01:59 NetworkManager[2060]: <info> (eth6): device state change: 3 -> 4 (reason 0) Sep 8 11:01:59 NetworkManager[2060]: <info> Activation (eth6) Stage 1 of 5 (Device Prepare) scheduled...
Sep 8 11:01:59 NetworkManager[2060]: <info> Activation (eth6) Stage 1 of 5 (Device Prepare) started...
Sep 8 11:01:59 NetworkManager[2060]: <info> Activation (eth6) Stage 2 of 5 (Device Configure) scheduled...
Sep 8 11:01:59 NetworkManager[2060]: <info> Activation (eth6) Stage 1 of 5 (Device Prepare) complete.
Sep 8 11:01:59 NetworkManager[2060]: <info> Activation (eth6) Stage 2 of 5 (Device Configure) starting...
Sep 8 11:01:59 NetworkManager[2060]: <info> (eth6): device state change: 4 -> 5 (reason 0) Sep 8 11:01:59 NetworkManager[2060]: <info> Activation (eth6) Stage 2 of 5 (Device Configure) successful.
Sep 8 11:01:59 NetworkManager[2060]: <info> Activation (eth6) Stage 3 of 5 (IP Configure Start) scheduled.
Sep 8 11:01:59 NetworkManager[2060]: <info> Activation (eth6) Stage 2 of 5 (Device Configure) complete.
Sep 8 11:01:59 NetworkManager[2060]: <info> Activation (eth6) Stage 3 of 5 (IP Configure Start) started...
Sep 8 11:01:59 NetworkManager[2060]: <info> (eth6): device state change: 5 -> 7 (reason 0) Sep 8 11:01:59 NetworkManager[2060]: <info> Activation (eth6) Stage 4 of 5 (IP4 Configure Get) scheduled...
Sep 8 11:01:59 NetworkManager[2060]: <info> Activation (eth6) Stage 3 of 5 (IP Configure Start) complete.
Sep 8 11:01:59 NetworkManager[2060]: <info> Activation (eth6) Stage 4 of 5 (IP4 Configure Get) started...
Sep 8 11:01:59 NetworkManager[2060]: <info> Scheduling stage 5 Sep 8 11:01:59 NetworkManager[2060]: <info> Activation (eth6) Stage 5 of 5 (IP Configure Commit) scheduled...
Sep 8 11:01:59 NetworkManager[2060]: <info> Done scheduling stage 5 Sep 8 11:01:59 NetworkManager[2060]: <info> Activation (eth6) Stage 4 of 5 (IP4 Configure Get) complete.
Sep 8 11:01:59 NetworkManager[2060]: <info> Activation (eth6) Stage 5 of 5 (IP Configure Commit) started...
Sep 8 11:01:59 avahi-daemon[2073]: Joining mDNS multicast group on interface eth6.IPv4 with address 10.0.0.2.
Sep 8 11:01:59 avahi-daemon[2073]: New relevant interface eth6.IPv4 for mDNS.
Sep 8 11:01:59 avahi-daemon[2073]: Registering new address record for 10.0.0.2 on eth6.IPv4.
Sep 8 11:02:00 kernel: [260641.389586] ixgbe: Multiple Queue Support Disabled Sep 8 11:02:00 kernel: [260641.389593] ixgbe: 0000:0f:00.1: ixgbe_check_options: RSS is not supported while multiple queues are disabled. Disabling RSS.
Sep 8 11:02:00 kernel: [260641.389600] ixgbe: 0000:0f:00.1: ixgbe_check_options: Flow Director is not supported while multiple queues are disabled. Disabling Flow Director Sep 8 11:02:00 kernel: [260641.389618] ixgbe: 0000:0f:00.1: ixgbe_check_options: FCoE Offload feature enabled Sep 8 11:02:00 kernel: [260641.389661] ixgbe 0000:0f:00.1: (unregistered net_device): FCoE offload feature is not available. Disabling FCoE offload feature Sep 8 11:02:00 kernel: [260641.414340] ixgbe 0000:0f:00.1: (PCI Express:5.0GT/s:Width x8) 00:e0:ed:ff:60:05 Sep 8 11:02:00 kernel: [260641.414432] ixgbe 0000:0f:00.1: eth7: MAC: 2, PHY: 1, PBA No: FFFFFF-0FF Sep 8 11:02:00 kernel: [260641.414443] ixgbe 0000:0f:00.1: eth7: Enabled Features: RxQ: 1 TxQ: 1 RSC Sep 8 11:02:00 kernel: [260641.414464] ixgbe 0000:0f:00.1: eth7: Intel(R) 10 Gigabit Network Connection Sep 8 11:02:00 NetworkManager[2060]: <info> Policy set 'System eth0' (eth0) as default for IPv4 routing and DNS.
Sep 8 11:02:00 NetworkManager[2060]: <info> (eth6): device state change: 7 -> 8 (reason 0) Sep 8 11:02:00 NetworkManager[2060]: <info> Activation (eth6) successful, device activated.
Sep 8 11:02:00 NetworkManager[2060]: <info> Activation (eth6) Stage 5 of 5 (IP Configure Commit) complete.
Sep 8 11:02:00 NetworkManager[2060]: <info> (eth7): carrier is OFF Sep 8 11:02:00 NetworkManager[2060]: <info> (eth7): new Ethernet device (driver: 'ixgbe' ifindex: 53) Sep 8 11:02:00 NetworkManager[2060]: <info> (eth7): exported as /org/freedesktop/NetworkManager/Devices/51
Sep 8 11:02:00 NetworkManager[2060]: <info> (eth7): now managed Sep 8 11:02:00 NetworkManager[2060]: <info> (eth7): device state change: 1 -> 2 (reason 2) Sep 8 11:02:00 NetworkManager[2060]: <info> (eth7): bringing up device.
Sep 8 11:02:00 NetworkManager[2060]: <info> (eth7): preparing device.
Sep 8 11:02:00 NetworkManager[2060]: <info> (eth7): deactivating device (reason: 2).
Sep 8 11:02:00 kernel: [260641.769646] ADDRCONF(NETDEV_UP): eth7: link is not ready Sep 8 11:02:01 avahi-daemon[2073]: Registering new address record for fe80::2e0:edff:feff:6004 on eth6.*.
Sep 8 11:02:05 NetworkManager[2060]: <info> (eth6): carrier now OFF (device state 8, deferring action for 4 seconds) Sep 8 11:02:05 kernel: [260646.629042] ixgbe 0000:0f:00.0: eth6: detected SFP+: 5 Sep 8 11:02:05 kernel: [260646.851639] ixgbe 0000:0f:00.0: eth6: detected SFP+: 5 Sep 8 11:02:05 kernel: [260647.090225] ixgbe 0000:0f:00.0: eth6: NIC Link is Up 10 Gbps, Flow Control: TX Sep 8 11:02:05 NetworkManager[2060]: <info> (eth6): carrier now ON (device state 8) Sep 8 11:02:16 kernel: [260657.874831] device eth6 entered promiscuous mode Sep 8 11:03:11 kernel: [260712.799174] device eth6 left promiscuous mode Sep 8 11:06:50 kernel: [260930.664527] device eth6 entered promiscuous mode Sep 8 11:07:48 kernel: [260988.578813] device eth6 left promiscuous mode
Next I ran the same app but with no options passed to the ixgbe driver, so it defaulted to 64 queues; this is a 32 processor machine with Hyper Threading on.
I managed to get to 180 Megabits/s before packets were dropped.
Next I turned off Hyper Threading and then ran the driver with the default arguments; this doesn't seem to make any difference, if anything performance was slightly poorer than with it on.
Is this what you would have expected for the standard driver? Does it seem like its operating correctly?
Andrew
------------------------------------------------------------------------------
Why Cloud-Based Security and Archiving Make Sense Osterman Research conducted this study that outlines how and why cloud computing security and archiving is rapidly being adopted across the IT space for its ease of implementation, lower cost, and increased reliability. Learn more. http://www.accelacomm.com/jaw/sfnl/114/51425301/
_______________________________________________
E1000-devel mailing list
E10...@li...
https://lists.sourceforge.net/lists/listinfo/e1000-devel
To learn more about Intel® Ethernet, visit http://communities.intel.com/community/wired
|
