Menu
▾
▴
Re: [E1000-devel] kernel PANIC when attached to older equipment
|
From: Alexander D. <ale...@gm...> - 2016-10-13 20:09:40
|
On Wed, Oct 12, 2016 at 2:53 PM, John Osswald <joh...@ta...> wrote: > We have multiple systems (and system types) experiencing a kernel panic, > when they upgrade to Debian 8. We tried the native Debian IGB driver, as > well as, IGB 5.3.5.3, with no difference in behavior. The common element is > that they are attached to our Portwell CAR-3000, running Linux 2.6.26 and > e1000e: Intel(R) PRO/1000 Network Driver - 1.0.2.5-NAPI. > The CAR-3000 board from Portwell has one Intel Core 2 Duo E8400 CPU running > at 3.00 GHz, 4GB of memory, and an Intel 82574L controller. > > The setup is pretty simple, two nodes ( CAR-3000 and the DUT), with one > cable between them, running iperf. The DUT connection is through ports > attached to an I354. The PANICs have not been been observed, when using a > DUT's I210 ports. > > > During the test, the CAR-3000 floods pause frames back to the DUT. The > CAR-3000 sees frequent unit hangs and, itself, may PANIC. > > These issues are not seen, if autoneg is disabled on the CAR-3000 port > > > Oct 5 22:09:04 CAR-3000kernel: [25609.890202] e1000e: eth2 NIC Link is Down > Oct 5 22:09:04 CAR-3000kernel: [25609.894366] 0000:05:00.0: eth2: Detected > Tx Unit Hang: *(this is the port connected to the 7573)* > Oct 5 22:09:04 CAR-3000kernel: [25609.894367] TDH <48> > Oct 5 22:09:04 CAR-3000kernel: [25609.894368] TDT <92> > Oct 5 22:09:04 CAR-3000kernel: [25609.894368] next_to_use <92> > Oct 5 22:09:04 CAR-3000kernel: [25609.894369] next_to_clean <48> > Oct 5 22:09:04 CAR-3000kernel: [25609.894369] buffer_info[next_to_clean]: > Oct 5 22:09:04 CAR-3000kernel: [25609.894370] time_stamp > <10060025e> > Oct 5 22:09:04 CAR-3000kernel: [25609.894370] next_to_watch <48> > Oct 5 22:09:04 CAR-3000kernel: [25609.894371] jiffies > <100600b8a> > Oct 5 22:09:04 CAR-3000kernel: [25609.894372] next_to_watch.status <0> > Oct 5 22:09:07 CAR-3000kernel: [25613.074687] e1000e: eth2 NIC Link is Up > 1000 Mbps Full Duplex, Flow Control: RX/TX > > > Coincidental with the above Unit Hangs on the CAR-3000, the DUT may PANIC > > In this instance, The PANIC is occurred on a Lannar 7573 (connection > through an I354), however it has been seen on other platforms from different > vendors. > > > [ 110.986096] ------------[ cut here ]------------ > [ 110.990774] WARNING: CPU: 7 PID: 53 at net/sched/sch_generic.c:264 > dev_watchdog+0xd9/0x13f() > [ 110.999300] NETDEV WATCHDOG: tn-lbp0 (igb): transmit queue 0 timed out > [ 111.005879] Modules linked in: dmi_sysfs ipv6 fuse coretemp kvm_intel kvm > crc32c_intel aesni_intel aes_x86_64 ablk_helper cryptd lrw gf128mul > glue_helper iTCO_wdt iTCO_vendor_support i2c_i801 microcode i2c_core lpc_ich > mfd_core ipmi_msghandler button ext3 mbcache jbd sd_mod crc_t10dif > crct10dif_generic crct10dif_common ahci ehci_pci libahci ehci_hcd igb(O) > libata dca ptp usbcore scsi_mod usb_common pps_core > [ 111.043335] CPU: 7 PID: 53 Comm: ksoftirqd/7 Tainted: G O > 3.16.7-e100v1 #1 > [ 111.051218] Hardware name: To be filled by O.E.M. To be filled by > O.E.M./To be filled by O.E.M., BIOS 5.6.5 06/30/2016 > [ 111.061966] 0000000000000006 ffffffff812fefff ffff88046ed03d20 > 0000000000000009 > [ 111.069559] ffffffff8103a0a9 ffffffff8128dec0 ffff88046e410000 > ffff88046ed03d78 > [ 111.077122] ffffffff8128dde7 ffff88046e410388 ffffffff8103a102 > ffffffff8159d053 > [ 111.084686] Call Trace: > [ 111.087147] [<ffffffff812fefff>] ? dump_stack+0x46/0x59 > [ 111.092497] [<ffffffff8103a0a9>] ? warn_slowpath_common+0x74/0x89 > [ 111.098731] [<ffffffff8128dec0>] ? dev_watchdog+0xd9/0x13f > [ 111.104334] [<ffffffff8128dde7>] ? dev_deactivate_queue+0x53/0x53 > [ 111.110545] [<ffffffff8103a102>] ? warn_slowpath_fmt+0x44/0x4d > [ 111.116495] [<ffffffff81302a8e>] ? _raw_spin_lock+0xe/0x17 > [ 111.122097] [<ffffffff81302b05>] ? _raw_spin_unlock+0xe/0x20 > [ 111.127873] [<ffffffff8128db84>] ? netif_tx_lock+0x6c/0x79 > [ 111.133491] [<ffffffff8128dec0>] ? dev_watchdog+0xd9/0x13f > [ 111.139098] [<ffffffff81042ebb>] ? call_timer_fn+0x36/0x108 > [ 111.144787] [<ffffffff8128dde7>] ? dev_deactivate_queue+0x53/0x53 > [ 111.150998] [<ffffffff81043785>] ? run_timer_softirq+0x1dd/0x20a > [ 111.157122] [<ffffffff8103d8f9>] ? __do_softirq+0xf8/0x27d > [ 111.162734] [<ffffffff8103da93>] ? run_ksoftirqd+0x15/0x4d > [ 111.168362] [<ffffffff810582f5>] ? smpboot_thread_fn+0x207/0x226 > [ 111.174521] [<ffffffff810580ee>] ? SyS_setgroups+0xe3/0xe3 > [ 111.180149] [<ffffffff81053223>] ? kthread+0xa8/0xb0 > [ 111.185224] [<ffffffff8105317b>] ? __kthread_parkme+0x5b/0x5b > [ 111.191105] [<ffffffff81303148>] ? ret_from_fork+0x58/0x90 > [ 111.196717] [<ffffffff8105317b>] ? __kthread_parkme+0x5b/0x5b > [ 111.202604] ---[ end trace 0242955cf46c53c7 ]--- > > > Thank you The error messages you are seeing indicate that the Tx isn't moving. If the CAR-3000 is flooding pause frames this is somewhat expected behavior. The warnings mean that the Tx path is full of packets and that the Tx queues are not draining. By the way the "panic" you refer to is actually a netdev watchdog warning. It has the same root cause. The stack and the driver think the Tx is hung because DMA is not moving so it is resetting the device. If you are wanting to work around this you could disable flow control on the interface using "ethtool -A <iface> tx off rx off autoneg off" and the NIC will ignore the hangs on the switch. - Alex |
