Menu
▾
▴
Re: [E1000-devel] 82599 latency increase with rx_missed_errors
|
From: Skidmore, D. C <don...@in...> - 2013-04-30 22:10:50
|
Hi Nishit,
The rx_no_dma_resources means we are dropping packets because we don't have any free descriptors in the RX Queue. While rx_missed_errors are due to insufficient space to store an ingress packet, so basically we ran out of buffers or bandwidth on the PCIe bus.
All that said when you see the rx_no_dma_resources errors is their rate comparable with what you were seeing for rx_missed_errors? Both will lead to the same thing, dropped packets.
Also what does 'lspci -vvv' show, I'm looking to see if you are getting the full PCIe bandwidth. You could also try to turn on FC which should lower these types of overflow occurrences.
Thanks,
-Don Skidmore <don...@in...>
> -----Original Message-----
> From: Nishit Shah [mailto:nis...@el...]
> Sent: Tuesday, April 30, 2013 9:07 AM
> To: e10...@li...
> Subject: [E1000-devel] 82599 latency increase with rx_missed_errors
>
>
> Hi,
>
> We are measuring packet latencies at various packet sizes (64 bytes to
> 1518 bytes) with 82599 card with ixgbe driver 3.7.21.
>
> Setup:
>
> Spirent test center sender machine with 82599
> (ixgbe 3.7.21 and vanilla 2.6.39.4) Spirent test center receiver
>
> 10 G <------------------------> 10G
> 10G <------------------------------> 10G
>
> When we don't have an increase in "rx_missed_errors" and
> "rx_no_dma_resources", we are getting per packet latency around 40-70
> microseconds. ("rx_no_buffer_count" is not increasing)
> When we have an increase in "rx_no_dma_resources", we are still getting
> per packet latency around 40-70 microseconds.
> ("rx_no_buffer_count" is not increasing)
> When we have an increase in "rx_missed_errors", we are getting per
> packet latency around 500 microseconds. (rx_no_buffer_count is not
> increasing)
>
> Is there any specific reason for latency increase when "rx_missed_errors"
> are increased ?
> Is there a way to control it ?
>
> Below is a machine detail.
> ==========================================================
> ===============================================
> Machine details.
>
> CPU: Dual Core Intel(R) Celeron(R) CPU G540 @ 2.50GHz
> Memory: 2 GB
> kernel: vanilla 2.6.39.4
> Interface tuning parameters:
> Auto Negotiation is off (DROP_EN is set.)
> ethtool -G eth0 rx 64 tx 128 ; ethtool -G eth1 rx 64 tx 128
> rx-usecs is set to 50.
> ethtool and lspci for bus information:
>
> # ethtool -i eth0
> driver: ixgbe
> version: 3.7.21-NAPI
> firmware-version: 0x80000345
> bus-info: 0000:06:00.0
> #
> # ethtool -i eth1
> driver: ixgbe
> version: 3.7.21-NAPI
> firmware-version: 0x80000345
> bus-info: 0000:06:00.1
>
> 06:00.0 Class 0200: Device 8086:10fb (rev 01)
> Subsystem: Device 15bb:30e0
> Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
> ParErr- Stepping- SERR- FastB2B- DisINTx+
> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast
> >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
> Latency: 0, Cache Line Size: 64 bytes
> Interrupt: pin A routed to IRQ 18
> Region 0: Memory at f7520000 (64-bit, non-prefetchable) [size=128K]
> Region 2: I/O ports at 8020 [size=32]
> Region 4: Memory at f7544000 (64-bit, non-prefetchable) [size=16K]
> Capabilities: [40] Power Management version 3
> Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA
> PME(D0+,D1-,D2-,D3hot+,D3cold-)
> Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
> Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
> Address: 0000000000000000 Data: 0000
> Masking: 00000000 Pending: 00000000
> Capabilities: [70] MSI-X: Enable+ Count=64 Masked-
> Vector table: BAR=4 offset=00000000
> PBA: BAR=4 offset=00002000
> Capabilities: [a0] Express (v2) Endpoint, MSI 00
> DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <512ns, L1
> <64us
> ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
> DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
> Unsupported-
> RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-
> MaxPayload 128 bytes, MaxReadReq 512 bytes
> DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+
> AuxPwr- TransPend-
> LnkCap: Port #1, Speed 5GT/s, Width x8, ASPM L0s, Latency L0 <2us,
> L1 <32us
> ClockPM- Surprise- LLActRep- BwNot-
> LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain-
> CommClk-
> ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
> LnkSta: Speed 5GT/s, Width x8, TrErr- Train- SlotClk+
> DLActive- BWMgmt- ABWMgmt-
> DevCap2: Completion Timeout: Range ABCD, TimeoutDis+
> DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
> LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-,
> Selectable De-emphasis: -6dB
> Transmit Margin: Normal Operating Range,
> EnterModifiedCompliance- ComplianceSOS-
> Compliance De-emphasis: -6dB
> LnkSta2: Current De-emphasis Level: -6dB
> Capabilities: [e0] Vital Product Data
> Unknown small resource type 06, will not decode more.
> Kernel driver in use: ixgbe
> Kernel modules: ixgbe
>
> 06:00.1 Class 0200: Device 8086:10fb (rev 01)
> Subsystem: Device 15bb:30e0
> Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
> ParErr- Stepping- SERR- FastB2B- DisINTx+
> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast
> >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
> Latency: 0, Cache Line Size: 64 bytes
> Interrupt: pin B routed to IRQ 19
> Region 0: Memory at f7500000 (64-bit, non-prefetchable) [size=128K]
> Region 2: I/O ports at 8000 [size=32]
> Region 4: Memory at f7540000 (64-bit, non-prefetchable) [size=16K]
> Capabilities: [40] Power Management version 3
> Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA
> PME(D0+,D1-,D2-,D3hot+,D3cold-)
> Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
> Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
> Address: 0000000000000000 Data: 0000
> Masking: 00000000 Pending: 00000000
> Capabilities: [70] MSI-X: Enable+ Count=64 Masked-
> Vector table: BAR=4 offset=00000000
> PBA: BAR=4 offset=00002000
> Capabilities: [a0] Express (v2) Endpoint, MSI 00
> DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <512ns, L1
> <64us
> ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
> DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
> Unsupported-
> RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-
> MaxPayload 128 bytes, MaxReadReq 512 bytes
> DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+
> AuxPwr- TransPend-
> LnkCap: Port #1, Speed 5GT/s, Width x8, ASPM L0s, Latency L0 <2us,
> L1 <32us
> ClockPM- Surprise- LLActRep- BwNot-
> LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain-
> CommClk-
> ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
> LnkSta: Speed 5GT/s, Width x8, TrErr- Train- SlotClk+
> DLActive- BWMgmt- ABWMgmt-
> DevCap2: Completion Timeout: Range ABCD, TimeoutDis+
> DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
> LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-,
> Selectable De-emphasis: -6dB
> Transmit Margin: Normal Operating Range,
> EnterModifiedCompliance- ComplianceSOS-
> Compliance De-emphasis: -6dB
> LnkSta2: Current De-emphasis Level: -6dB
> Capabilities: [e0] Vital Product Data
> Unknown small resource type 06, will not decode more.
> Kernel driver in use: ixgbe
> Kernel modules: ixgbe
> ==========================================================
> ===============================================
>
> Rgds,
> Nishit Shah.
>
> ------------------------------------------------------------------------------
> Introducing AppDynamics Lite, a free troubleshooting tool for Java/.NET Get
> 100% visibility into your production application - at no cost.
> Code-level diagnostics for performance bottlenecks with <2% overhead
> Download for free and get started troubleshooting in minutes.
> http://p.sf.net/sfu/appdyn_d2d_ap1
> _______________________________________________
> E1000-devel mailing list
> E10...@li...
> https://lists.sourceforge.net/lists/listinfo/e1000-devel
> To learn more about Intel® Ethernet, visit
> http://communities.intel.com/community/wired
|