Re: [E1000-devel] getting max throughput from AMD Opteron w/igb-2.2.9

[Ed R. <er...@pa...>]

On Tue, Jul 27, 2010 at 09:42:16AM -0700, Alexander Duyck wrote:

> The fact that the ring size seems to effect the number of packets  
> dropped per second implies that there may be some sort of latency issue.  
>  One thing you might try is different values for rx-usecs via ethtool  
> -C.  You may find that fixing the value at something fairly low like 33  
> usecs per interrupt may help to reduce the number of rx_fifo_errors.

"ethtool -C eth0 rx-usecs 33" is accepted, but "ethtool -c eth0" shows
the values unchanged.  This is with igb-2.2.9.


> After looking over your lspci dump I am assuming you are running a  
> Supermicro motherboard with the AMD SR5690/SP5100 chipset.  If that is  
> the case you will probably find that one physical ID works much better  
> than the other for network performance because the SR5690 that the 82576  
> is connected to is going to be node local for one of the sockets and  
> remote for the other.
>
> Another factor you will need to take into account is that the ring  
> memory should be allocated on the same node the hardware is on.  You  
> should be able to accomplish that by using taskset with the correct CPU  
> mask for the physical ID you are using when calling modprobe/insmod and  
> the ifconfig commands to bring up the interfaces.  This should help to  
> decrease the memory latency and increase the throughput available to the  
> adapter.

The taskset/modprobe trick along with putting all the queues on the same
physical CPU seems to provide the best performance, when using these
igb-2.2.9 settings:

 taskset 0002 modprobe igb RSS=0,0 InterruptThrottleRate=3,3

 root@big-tester:~# eth_affinity_tool show eth0 eth1
 16 CPUs detected

 eth0:  ffff 0001 0002 0004 0008 0010 0020 0040 0080
 eth1:  ffff 0001 0002 0004 0008 0010 0020 0040 0080

> Thanks for the information.  One other item I would be interested in  
> seeing is the kind of numbers we are talking about.  If you could  
> provide me with an ethtool -S dump from 10 seconds of one of your tests  
> that might be useful for me to better understand the kind of pressures  
> the system is under.

Here's a sample 10-second run of "ethtool -S" on the receiving interface,
after being piped through beforeafter, so these are 10 seconds of each
counter.  Counters that are missing were zero, i.e., no changes in the
10 second run.  So dividing these numbers by 10 gives you the per-second
rate.

In this interval, we're sending far more packets than
can be processed:

Interval from 20100727.174619 to 20100727.174629

NIC statistics:
     rx_packets: 11595293
     rx_bytes: 742098688
     rx_long_byte_count: 742098688
     rx_fifo_errors: 163800
     rx_queue_0_packets: 754216
     rx_queue_0_bytes: 45252960
     rx_queue_0_drops: 20475
     rx_queue_1_packets: 760734
     rx_queue_1_bytes: 45644040
     rx_queue_1_drops: 20475
     rx_queue_2_packets: 736546
     rx_queue_2_bytes: 44192760
     rx_queue_2_drops: 20475
     rx_queue_3_packets: 742368
     rx_queue_3_bytes: 44542080
     rx_queue_3_drops: 20475
     rx_queue_4_packets: 661758
     rx_queue_4_bytes: 39705480
     rx_queue_4_drops: 20475
     rx_queue_5_packets: 713095
     rx_queue_5_bytes: 42785706
     rx_queue_5_drops: 20475
     rx_queue_6_packets: 696702
     rx_queue_6_bytes: 41802120
     rx_queue_6_drops: 20475
     rx_queue_7_packets: 705726
     rx_queue_7_bytes: 42343560
     rx_queue_7_drops: 20475

And in this interval, we're sending packets just a bit faster than they
can be processed:

Interval from 20100727.175747 to 20100727.175757

NIC statistics:
     rx_packets: 3877553
     rx_bytes: 248163392
     rx_long_byte_count: 248163392
     rx_fifo_errors: 6515
     rx_queue_0_packets: 484608
     rx_queue_0_bytes: 29076480
     rx_queue_0_drops: 81
     rx_queue_1_packets: 484690
     rx_queue_1_bytes: 29081400
     rx_queue_2_packets: 484690
     rx_queue_2_bytes: 29081400
     rx_queue_3_packets: 484360
     rx_queue_3_bytes: 29061600
     rx_queue_3_drops: 342
     rx_queue_4_packets: 483207
     rx_queue_4_bytes: 28992420
     rx_queue_4_drops: 1497
     rx_queue_5_packets: 480183
     rx_queue_5_bytes: 28810980
     rx_queue_5_drops: 4521
     rx_queue_6_packets: 484615
     rx_queue_6_bytes: 29076900
     rx_queue_6_drops: 74
     rx_queue_7_packets: 484690
     rx_queue_7_bytes: 29081400

Again, to keep things readable, my counter-processing script doesn't
list statistics that didn't increment during the 10 second interval,
so the stats from "ethtool -S" not listed were all zero.

Note that the traffic I'm using to get these numbers (lots of small UDP
packets)  is a denial-of-service scenario - we're more interested
in real-world performance for routing but that's harder to simulate, and
we need to be able to handle DoS attacks so this is the benchmark we're
using.

Thanks,

	-- Ed