#382 eth0 (e1000e): transmit queue 0 timed out

open
None
in-kernel_driver
1
2015-08-20
2013-10-23
efrancone
No

I have a problem with e1000e centos 5.9 kernel 2.6.32-358, ethernet crash and the only solution is a reboot.

----------------------MOTHER-------------------
BIOS Information
Vendor: Intel Corp.
Version: S1200BT.86B.02.00.0041.120520121743

----------------CALL TRACE----------------------

Oct 23 08:53:00 rtrpc kernel: WARNING: at net/sched/sch_generic.c:261 dev_watchdog+0x263/0x270() (Not tainted)
Oct 23 08:53:00 rtrpc kernel: Hardware name: SandyBridge Platform
Oct 23 08:53:00 rtrpc kernel: NETDEV WATCHDOG: eth0 (e1000e): transmit queue 0 timed out
Oct 23 08:53:00 rtrpc kernel: Modules linked in: cls_u32(U) sch_sfq(U) sch_htb(U) ipt_MASQUERADE(U) ipt_LOG(U) xt_tcpudp(U) xt_state(U) xt_recent(U) iptable_filter(U)
iptable_nat(U) xt_layer7(U) xt_IMQ(U) xt_MARK(U) xt_CONNMARK(U) xt_mark(U) xt_multiport(U) iptable_mangle(U) ip_tables(U) nf_nat_tftp(U) nf_nat_sip(U) nf_nat_pptp(U) n
f_nat_proto_gre(U) nf_nat_irc(U) nf_nat_h323(U) nf_nat_amanda(U) nf_conntrack_tftp(U) nf_conntrack_sip(U) nf_conntrack_sane(U) nf_conntrack_proto_sctp(U) nf_conntrack_
pptp(U) nf_conntrack_proto_gre(U) nf_conntrack_netlink(U) nfnetlink(U) nf_conntrack_netbios_ns(U) nf_conntrack_broadcast(U) nf_conntrack_irc(U) nf_conntrack_h323(U) ts
_kmp(U) nf_conntrack_amanda(U) nf_nat_ftp(U) nf_nat(U) nf_conntrack_ipv4(U) nf_defrag_ipv4(U) nf_conntrack_ftp(U) nf_conntrack(U) xt_ipp2p(U) compat_xtables(U) ip6_tab
les(U) x_tables(U) imq(U) ipv6(U) dm_mirror(U) dm_multipath(U) sbs(U) sbshc(U) power_meter(U) acpi_pad(U) parport_pc(U) lp(U) parport(U) video(U) output(U) sg(U) sr_mo
d(U) cdrom(U) e1000e(U)
Oct 23 08:53:00 rtrpc kernel: i2c_i801(U) i2c_core(U) iTCO_wdt(U) iTCO_vendor_support(U) shpchp(U) dm_raid45(U) dm_memcache(U) xor(U) dm_region_hash(U) dm_log(U) dm_mo
d(U) ide_gd_mod(U) ata_piix(U) ahci(U) libata(U) ext3(U) jbd(U) mbcache(U) uhci_hcd(U) ohci_hcd(U) ehci_hcd(U) [last unloaded: microcode]
Oct 23 08:53:00 rtrpc kernel: Pid: 0, comm: swapper Not tainted 2.6.32-358.18.1.el6-1550AB_SOPNET #1
rtrpc kernel: Call Trace:
rtrpc kernel: <IRQ> [<ffffffff81434e93>] ? dev_watchdog+0x263/0x270
Oct 23 08:53:00 rtrpc kernel: [<ffffffff81434e93>] ? dev_watchdog+0x263/0x270
Oct 23 08:53:00 rtrpc kernel: [<ffffffff81069ca8>] ? warn_slowpath_common+0x98/0xc0
Oct 23 08:53:00 rtrpc kernel: [<ffffffff81069dce>] ? warn_slowpath_fmt+0x6e/0x70
Oct 23 08:53:00 rtrpc kernel: [<ffffffff81448f20>] ? ip_rcv_finish+0x0/0x420
Oct 23 08:53:00 rtrpc kernel: [<ffffffff8126904f>] ? strlcpy+0x4f/0x70
Oct 23 08:53:00 rtrpc kernel: [<ffffffff81413468>] ? netdev_drivername+0x48/0x60
Oct 23 08:53:00 rtrpc kernel: [<ffffffff81434e93>] ? dev_watchdog+0x263/0x270
Oct 23 08:53:00 rtrpc kernel: [<ffffffff81057516>] ? scheduler_tick+0xe6/0x240
Oct 23 08:53:00 rtrpc kernel: [<ffffffff81434c30>] ? dev_watchdog+0x0/0x270
Oct 23 08:53:00 rtrpc kernel: [<ffffffff8107cb39>] ? run_timer_softirq+0x169/0x300
Oct 23 08:53:00 rtrpc kernel: [<ffffffff8102d63d>] ? lapic_next_event+0x1d/0x30
Oct 23 08:53:00 rtrpc kernel: [<ffffffff810725e0>] ? __do_softirq+0xc0/0x1e0
Oct 23 08:53:00 rtrpc kernel: [<ffffffff8100c0cc>] ? call_softirq+0x1c/0x30
Oct 23 08:53:00 rtrpc kernel: [<ffffffff8100dc95>] ? do_softirq+0x65/0xa0
Oct 23 08:53:00 rtrpc kernel: [<ffffffff8107249c>] ? irq_exit+0x7c/0x90
Oct 23 08:53:00 rtrpc kernel: [<ffffffff814e2210>] ? smp_apic_timer_interrupt+0x70/0x9d
Oct 23 08:53:00 rtrpc kernel: [<ffffffff8100ba93>] ? apic_timer_interrupt+0x13/0x20
Oct 23 08:53:00 rtrpc kernel: <EOI> [<ffffffff812bb1ae>] ? intel_idle+0xde/0x160
Oct 23 08:53:00 rtrpc kernel: [<ffffffff812bb191>] ? intel_idle+0xc1/0x160
Oct 23 08:53:00 rtrpc kernel: [<ffffffff813e29e9>] ? cpuidle_idle_call+0x99/0x130
Oct 23 08:53:00 rtrpc kernel: [<ffffffff81009808>] ? cpu_idle+0xa8/0xe0
Oct 23 08:53:00 rtrpc kernel: [<ffffffff814d1096>] ? start_secondary+0x226/0x360
Oct 23 08:53:00 rtrpc kernel: ---[ end trace 43b6b9b7fbb38c66 ]---
Oct 23 08:53:00 rtrpc kernel: e1000e 0000:00:19.0: eth0: Reset adapter
Oct 23 08:53:02 rtrpc kernel: e1000e: eth0 NIC Link is Up 100 Mbps Full Duplex, Flow Control: Rx/Tx
Oct 23 08:53:02 rtrpc kernel: e1000e 0000:00:19.0: eth0: 10/100 speed: disabling TSO
Oct 23 08:53:12 rtrpc kernel: e1000e 0000:00:19.0: eth0: Reset adapter


ethtool -i eth0
driver: e1000e
version: 2.1.4-k
firmware-version: 0.13-4
bus-info: 0000:00:19.0


lspci -vvv -nn -d 8086:1502
00:19.0 Ethernet controller [0200]: Intel Corporation 82579LM Gigabit Network Connection [8086:1502] (rev 05)
Subsystem: Intel Corporation Device [8086:357a]
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort-="">SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 30
Region 0: Memory at c1b00000 (32-bit, non-prefetchable) [size=128K]
Region 1: Memory at c1b70000 (32-bit, non-prefetchable) [size=4K]
Region 2: I/O ports at 4040 [size=32]
Capabilities: [c8] Power Management version 2
Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [d0] MSI: Enable+ Count=1/1 Maskable- 64bit+
Address: 00000000fee07000 Data: 4082
Capabilities: [e0] PCI Advanced Features
AFCap: TP+ FLR+
AFCtrl: FLR-
AFStatus: TP-
Kernel driver in use: e1000e
Kernel modules: e1000e

Discussion

  • Todd Fujinaka

    Todd Fujinaka - 2013-12-03
    • assigned_to: dertman
     
  • Nicolas Krzywinski

    Hello, I have exactly the same problem on debian:


    Base Board Information
        Manufacturer: TYAN
        Product Name: S7002
    

    $ uname -a
    Linux server7even3 3.2.0-4-amd64 #1 SMP Debian 3.2.51-1 x86_64 GNU/Linux
    

    $ lsb_release --description
    Description:    Debian GNU/Linux 7.2 (wheezy)
    

    Dec 11 20:23:34 server7even3 kernel: [611658.305046] WARNING: at /build/linux-rrsxby/linux-3.2.51/net/sched/sch_generic.c:256 dev_watchdog+0xf2/0x151()
    Dec 11 20:23:34 server7even3 kernel: [611658.305049] Hardware name: empty
    Dec 11 20:23:34 server7even3 kernel: [611658.305052] NETDEV WATCHDOG: eth0 (e1000e): transmit queue 0 timed out
    Dec 11 20:23:34 server7even3 kernel: [611658.305054] Modules linked in: ipheth tun ip6table_filter ip6_tables iptable_filter ip_tables ebtable_nat ebtables x_tables mperf cpufreq_userspace cpufreq_stats cpufreq_conservative cpufreq_powersave parport_pc ppdev lp parport rfcomm bnep bluetooth rfkill binfmt_misc fuse nfsd nfs nfs_acl auth_rpcgss fscache lockd sunrpc bridge stp ext3 jbd dm_crypt snd_dummy raid456 async_raid6_recov async_memcpy async_pq raid6_pq async_xor xor async_tx md_mod w83793 hwmon_vid loop kvm_intel kvm fglrx(P) snd_hda_codec_hdmi coretemp psmouse serio_raw pcspkr snd_hda_intel joydev snd_hda_codec evdev snd_hwdep snd_pcm snd_page_alloc snd_seq snd_seq_device snd_timer i2c_i801 iTCO_wdt snd i2c_core iTCO_vendor_support i7core_edac soundcore edac_core processor button thermal_sys ext4 crc16 jbd2 mbcache dm_mod sg sd_mod crc_t10dif usbhid hid ahci libahci crc32c_intel libata e1000e uhci_hcd ehci_hcd mptsas scsi_transport_sas mptscsih mptbase scsi_mod usbcore usb_common [last unloaded: scsi_wait_scan]
    Dec 11 20:23:34 server7even3 kernel: [611658.305131] Pid: 0, comm: swapper/0 Tainted: P           O 3.2.0-4-amd64 #1 Debian 3.2.51-1
    Dec 11 20:23:34 server7even3 kernel: [611658.305133] Call Trace:
    Dec 11 20:23:34 server7even3 kernel: [611658.305135]  <IRQ>  [<ffffffff81046cbd>] ? warn_slowpath_common+0x78/0x8c
    Dec 11 20:23:34 server7even3 kernel: [611658.305143]  [<ffffffff81046d69>] ? warn_slowpath_fmt+0x45/0x4a
    Dec 11 20:23:34 server7even3 kernel: [611658.305146]  [<ffffffff812a6f11>] ? netif_tx_lock+0x40/0x75
    Dec 11 20:23:34 server7even3 kernel: [611658.305150]  [<ffffffff812a7081>] ? dev_watchdog+0xf2/0x151
    Dec 11 20:23:34 server7even3 kernel: [611658.305154]  [<ffffffff81052480>] ? run_timer_softirq+0x19a/0x261
    Dec 11 20:23:34 server7even3 kernel: [611658.305158]  [<ffffffff8106b7fd>] ? tick_handle_oneshot_broadcast+0xbc/0xfa
    Dec 11 20:23:34 server7even3 kernel: [611658.305161]  [<ffffffff812a6f8f>] ? netif_tx_unlock+0x49/0x49
    Dec 11 20:23:34 server7even3 kernel: [611658.305164]  [<ffffffff8104c2f8>] ? __do_softirq+0xb9/0x177
    Dec 11 20:23:34 server7even3 kernel: [611658.305168]  [<ffffffff81023fdc>] ? __setup_APIC_LVTT+0x4a/0x66
    Dec 11 20:23:34 server7even3 kernel: [611658.305172]  [<ffffffff8135646c>] ? call_softirq+0x1c/0x30
    Dec 11 20:23:34 server7even3 kernel: [611658.305176]  [<ffffffff8100f8cd>] ? do_softirq+0x3c/0x7b
    Dec 11 20:23:34 server7even3 kernel: [611658.305179]  [<ffffffff8104c560>] ? irq_exit+0x3c/0x99
    Dec 11 20:23:34 server7even3 kernel: [611658.305181]  [<ffffffff8100f5fd>] ? do_IRQ+0x82/0x98
    Dec 11 20:23:34 server7even3 kernel: [611658.305185]  [<ffffffff8134f4ee>] ? common_interrupt+0x6e/0x6e
    Dec 11 20:23:34 server7even3 kernel: [611658.305187]  <EOI>  [<ffffffff811ee07d>] ? intel_idle+0xea/0x119
    Dec 11 20:23:34 server7even3 kernel: [611658.305193]  [<ffffffff811ee05c>] ? intel_idle+0xc9/0x119
    Dec 11 20:23:34 server7even3 kernel: [611658.305197]  [<ffffffff8126febd>] ? cpuidle_idle_call+0xec/0x179
    Dec 11 20:23:34 server7even3 kernel: [611658.305201]  [<ffffffff8100d243>] ? cpu_idle+0xa5/0xf2
    Dec 11 20:23:34 server7even3 kernel: [611658.305203]  [<ffffffff816abb36>] ? start_kernel+0x3b8/0x3c3
    Dec 11 20:23:34 server7even3 kernel: [611658.305206]  [<ffffffff816ab140>] ? early_idt_handlers+0x140/0x140
    Dec 11 20:23:34 server7even3 kernel: [611658.305209]  [<ffffffff816ab3c4>] ? x86_64_start_kernel+0x104/0x111
    Dec 11 20:23:34 server7even3 kernel: [611658.305211] ---[ end trace 5adbba59b752c2c0 ]---
    Dec 11 20:23:34 server7even3 kernel: [611658.305218] e1000e 0000:03:00.0: eth0: Reset adapter
    

    # ethtool -i eth0
    driver: e1000e
    version: 1.5.1-k
    firmware-version: 1.9-0
    bus-info: 0000:03:00.0
    supports-statistics: yes
    supports-test: yes
    supports-eeprom-access: yes
    supports-register-dump: yes
    supports-priv-flags: no
    

    # lspci -vvv -nn -d 8086:10d3
    02:00.0 Ethernet controller [0200]: Intel Corporation 82574L Gigabit Network Connection [8086:10d3]
        Subsystem: Intel Corporation Device [8086:0000]
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 5
        Region 0: Memory at fb9e0000 (32-bit, non-prefetchable) [size=128K]
        Region 2: I/O ports at bc00 [size=32]
        Region 3: Memory at fb9dc000 (32-bit, non-prefetchable) [size=16K]
        Capabilities: [c8] Power Management version 2
            Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
            Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
        Capabilities: [d0] MSI: Enable- Count=1/1 Maskable- 64bit+
            Address: 0000000000000000  Data: 0000
        Capabilities: [e0] Express (v1) Endpoint, MSI 00
            DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
                ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
            DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
                RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
                MaxPayload 128 bytes, MaxReadReq 512 bytes
            DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr+ TransPend-
            LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Latency L0 <128ns, L1 <64us
                ClockPM- Surprise- LLActRep- BwNot-
            LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes Disabled- Retrain- CommClk+
                ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
            LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
        Capabilities: [a0] MSI-X: Enable+ Count=5 Masked-
            Vector table: BAR=3 offset=00000000
            PBA: BAR=3 offset=00002000
        Capabilities: [100 v1] Advanced Error Reporting
            UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
            UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
            UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
            CESta:  RxErr+ BadTLP+ BadDLLP+ Rollover- Timeout+ NonFatalErr+
            CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
            AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
        Kernel driver in use: e1000e
    
    03:00.0 Ethernet controller [0200]: Intel Corporation 82574L Gigabit Network Connection [8086:10d3]
        Subsystem: Intel Corporation Device [8086:0000]
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 10
        Region 0: Memory at fbae0000 (32-bit, non-prefetchable) [size=128K]
        Region 2: I/O ports at cc00 [size=32]
        Region 3: Memory at fbadc000 (32-bit, non-prefetchable) [size=16K]
        Capabilities: [c8] Power Management version 2
            Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
            Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
        Capabilities: [d0] MSI: Enable- Count=1/1 Maskable- 64bit+
            Address: 0000000000000000  Data: 0000
        Capabilities: [e0] Express (v1) Endpoint, MSI 00
            DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
                ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
            DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
                RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
                MaxPayload 128 bytes, MaxReadReq 512 bytes
            DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr+ TransPend-
            LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Latency L0 <128ns, L1 <64us
                ClockPM- Surprise- LLActRep- BwNot-
            LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes Disabled- Retrain- CommClk+
                ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
            LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
        Capabilities: [a0] MSI-X: Enable+ Count=5 Masked-
            Vector table: BAR=3 offset=00000000
            PBA: BAR=3 offset=00002000
        Capabilities: [100 v1] Advanced Error Reporting
            UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
            UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
            UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
            CESta:  RxErr+ BadTLP+ BadDLLP+ Rollover+ Timeout+ NonFatalErr+
            CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
            AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
        Capabilities: [140 v1] Device Serial Number 00-e0-81-ff-ff-b7-42-f3
        Kernel driver in use: e1000e
    

    Is there a known workaround? Having to reboot the server is ... hard stuff :D
    I tried to disable power saving functions in BIOS, with no success.

     
  • Todd Fujinaka

    Todd Fujinaka - 2013-12-12
     
  • Todd Fujinaka

    Todd Fujinaka - 2013-12-12

    You do NOT have the exact same problem; you're not using the same part. Please open a new bug for this.

     
    • Nicolas Krzywinski

      part??
      I meant, the erroneous behaviour is exactly identical. But I can open a new bug if you want - unsure about how to specify this different part there!?

       
      • Todd Fujinaka

        Todd Fujinaka - 2013-12-13

        Original post: 82579LM
        Nicolas Krzywinski: 82574L

         
        • Nicolas Krzywinski

          Ah I see, its a different NIC ... or kind of.
          I will try to create a new bug report, thanks for your hint.

           
  • Todd Fujinaka

    Todd Fujinaka - 2015-05-12
    • assigned_to: dertman --> Yanir Lubetkin
     
  • Todd Fujinaka

    Todd Fujinaka - 2015-08-20
    • assigned_to: Yanir Lubetkin --> Raanan Avargil
     

Log in to post a comment.