#11 rx_missed_errors on X520-SR2 {E10G42BFSR}

closed
nobody
ixgbe (35)
standalone_driver
5
2014-07-29
2011-01-09
Evgen Bendyak
No

AMD two 8-core 6136 {OS6136WKT8EGO}
loading of modprobe ixgbe FdirMode=0,0
ethtool -G eth1 rx 4096
ethtool -G eth1 tx 4096
ethtool -K eth1 gso off
ethtool -K eth1 gro off
ethtool -K eth1 tso off
ethtool -A eth1 autoneg off rx off tx off

server used in forwarding packet for network
some time rx_missed_errors grows up 17000 packets/sec (on ping see drops around 2-3%)
rx_no_buffer_count still 0
rx_packets in that moment more 700000 packets/sec
on other port of Card eth0 in that moment 600000 packets/sec and rx_missed_errors not more 200-300 packets/sec.
as i read in data sheet on inet chip. MPC register is used for counting rx_missed_errors. In docs it looks like packets witch not put in internal chip packet buffer because it full. How it can happens. We use RSS queue connected to separate CPU. CPU load not more 40-50%. Is it hardware problem (maximum chip utilization) or can be software fixed.

Discussion

  • Evgen Bendyak
    Evgen Bendyak
    2011-01-10

    Found some additional info
    in driver you configure PFQDE (0x02F04; RW) QDE bit for each queue is off. also for each queue SRRCTL register you configured (in normal rss mode) without Drop_En bit. So (i think) main on chip packet buffer will stop receive new packet if one of RSS queue will not handle all packets and rx_no_buffer_count will not grow because drop not enabled. Am i righ?

     
  • Can you include an lspci -vvv dump for the system?

    Typically rx_missed_errors with no rx_no_buffer_count indicate that there is a problem in the PCIe bus bandwidth not being sufficient to allow for the packet DMA to memory. I recommend verifying that the PCIe slot being used is a x8 5.0GT/s slot in order to get the best performance.

     
  • Don Skidmore
    Don Skidmore
    2011-01-10

    It would also be interesting to see how the queue were distributed. The output from:

    cat /proc/interrupts

     
  • Evgen Bendyak
    Evgen Bendyak
    2011-01-10

    lspci -vvv

     
    Attachments
  • Evgen Bendyak
    Evgen Bendyak
    2011-01-10

    Driver report that it use a x8 5.0GT/s slot.
    about pci list see attach file

    for smp affinity i use such scrpit:

    echo 1 > /proc/irq/egrep -e 'eth0-TxRx-0$' /proc/interrupts | awk '{print $1}' | tr -d ':'/smp_affinity
    echo 2 > /proc/irq/egrep -e 'eth0-TxRx-1$' /proc/interrupts | awk '{print $1}' | tr -d ':'/smp_affinity
    echo 4 > /proc/irq/egrep -e 'eth0-TxRx-2$' /proc/interrupts | awk '{print $1}' | tr -d ':'/smp_affinity
    echo 8 > /proc/irq/egrep -e 'eth0-TxRx-3$' /proc/interrupts | awk '{print $1}' | tr -d ':'/smp_affinity
    echo 10 > /proc/irq/egrep -e 'eth0-TxRx-4$' /proc/interrupts | awk '{print $1}' | tr -d ':'/smp_affinity
    echo 20 > /proc/irq/egrep -e 'eth0-TxRx-5$' /proc/interrupts | awk '{print $1}' | tr -d ':'/smp_affinity
    echo 40 > /proc/irq/egrep -e 'eth0-TxRx-6$' /proc/interrupts | awk '{print $1}' | tr -d ':'/smp_affinity
    echo 80 > /proc/irq/egrep -e 'eth0-TxRx-7$' /proc/interrupts | awk '{print $1}' | tr -d ':'/smp_affinity
    echo 100 > /proc/irq/egrep -e 'eth0-TxRx-8$' /proc/interrupts | awk '{print $1}' | tr -d ':'/smp_affinity
    echo 200 > /proc/irq/egrep -e 'eth0-TxRx-9$' /proc/interrupts | awk '{print $1}' | tr -d ':'/smp_affinity
    echo 400 > /proc/irq/egrep -e 'eth0-TxRx-10$' /proc/interrupts | awk '{print $1}' | tr -d ':'/smp_affinity
    echo 800 > /proc/irq/egrep -e 'eth0-TxRx-11$' /proc/interrupts | awk '{print $1}' | tr -d ':'/smp_affinity
    echo 1000 > /proc/irq/egrep -e 'eth0-TxRx-12$' /proc/interrupts | awk '{print $1}' | tr -d ':'/smp_affinity
    echo 2000 > /proc/irq/egrep -e 'eth0-TxRx-13$' /proc/interrupts | awk '{print $1}' | tr -d ':'/smp_affinity
    echo 4000 > /proc/irq/egrep -e 'eth0-TxRx-14$' /proc/interrupts | awk '{print $1}' | tr -d ':'/smp_affinity
    echo 8000 > /proc/irq/egrep -e 'eth0-TxRx-15$' /proc/interrupts | awk '{print $1}' | tr -d ':'/smp_affinity

    the same for eth1-TxRx
    such script give that queue0=cpu0,queue1=cpu1 and so on to queue15=cpu15

    I found strange thing. I play some more with interrupts affinity with cpu and by sample setup queue0-queue7=cpu0-cpu7 queue8-queue15=cpu0-cpu7 (the same for both eth0 and eth1) and rx_missed_errors stop to grow and total speed on line grows (because no drops on line). I try to make other this eth0 queue0-queue15 use only cpu0-cpu7 and eth1 queue0-queue15 use only cpu8-cpu15 - result rx_missed_errors returns in same amount like was before. So it look like if all 16 cores at same moment work with pciex or may be problem in intel ethernet chip (it have not possibility to work with such amount of requests at same time). I checked this idea on 24 core system make 12 queue for each ethX and atatch it to different cores eth0 to cpu0-cpu11, eth1 to cpu12-cpu23 - result rx_missed_errors begins to grow much more earlier on lover amount of traffic begins from 4,4 Gigabit of rx. So what you thinks about such thing or may be need to ask intel about that. Also i have second such card and have other PCIex x8 2.0 slot and can test cards when only one of eth fro card is used for traffic. Maybe intel not plans to use both eth channels at same on one card. Also make static test of static trafic on small packets system grows up to 4,5 Mpps and eat all cpu but still rx_missed_errors not grow (so looks like problem in PCIex banwidth when both eth used and 16 cores also works with card) Thanks.

     
  • Jacob Keller
    Jacob Keller
    2012-05-04

    This issue is more than a year old. Is there still a problem, or has this been resolved? Please reply within 60 days if there is still an issue, or the bug will automatically be closed.

     
  • Evgen Bendyak
    Evgen Bendyak
    2012-05-05

    For this year i made a lot of tests on different platforms Intel & AMD. First of all on AMD opteron platform Max Payload Size on PCIExpres is 128 byte maximum (in datashete on ethernet chip) for two port usage recommended minimum 256 byte (intel server platform suport it). Also i think, on AMD platform need to use different nodes for each ethernet channel (amd have memory 4 nodes), not test more on amd platform. On intel server platform on same traffic it work normal. So i think its total platform problem that its not give needed bandwidth to ram. Ethernet chip write data to ram buffer but its not possible and go to wait state and begin to increment this counter. Its only my opinion only, but all looks like that.

     
  • Todd Fujinaka
    Todd Fujinaka
    2013-07-09

    • status: pending --> closed
    • assigned_to: Alexander Duyck --> nobody
     
  • Todd Fujinaka
    Todd Fujinaka
    2013-07-09

    Closed due to age.