FOSS-Cloud / Bugs / #270 Problem with NIC

#270 Problem with NIC

Milestone: v.1.2.10

Status: open

Owner: nobody

Labels: None

Component: Unknown

Priority: None

Updated: 2013-08-27

Created: 2013-07-26

Creator: Stefan Mühling

Private: No

Problem with NIC

Hardware: ProLiant BL460c G6

Problem with NIC (happend to both used blades randomly):
System freezes sometimes. No special situation when this occures. First it only happened when
the backup was started over night, but not always. Disabled automatic backup and
happened later without any traffic internally or externally.

Logfile is available at http://ftp.pdv-systeme.de/mu/messages-tar.gz (extracted size is about 3gb)
I hope this file contains the relevant information. I can't check at the moment.

I can't reproduce the error at will. If it occurs again I try to make a screenshot. Looks like a Kernel Panic produced by the NIC driver.

Discussion

Stefan Mühling - 2013-08-26

Here is the essential information from /var/log/messages

Aug 22 15:33:47 primary kernel: : [1489938.428057] ------------[ cut here ]------------
Aug 22 15:33:47 primary kernel: : [1489938.428070] WARNING: at net/core/dev.c:2029 skb_warn_bad_offload+0xb7/0xc0()
Aug 22 15:33:47 primary kernel: : [1489938.428073] Hardware name: ProLiant BL460c G6
Aug 22 15:33:47 primary kernel: : [1489938.428078] : caps=(0x0000000040004040, 0x0000000000000000) len=5880 data_len=4380 gso_size=1460 gso_type=1 ip_summed=1
Aug 22 15:33:47 primary kernel: : [1489938.428081] Modules linked in: iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi nfnetlink_log nfnetlink vhost_net macvtap macvlan gfs2 dlm sctp configfs ebtable_nat ebtables bridge stp llc bonding coretemp mperf freq_table crc32c_intel microcode bnx2x iTCO_wdt iTCO_vendor_support lpc_ich hpilo hpwdt mfd_core ehci_pci i7core_edac kvm_intel edac_core mdio acpi_power_meter button ablk_helper cryptd lrw xts gf128mul aes_x86_64 sha256_generic tg3 libphy hwmon ptp pps_core e1000 fuse nfs fscache lockd sunrpc reiserfs btrfs zlib_deflate lzo_compress ext4 jbd2 ext3 jbd dm_crypt dm_mirror dm_region_hash dm_log sl811_hcd xhci_hcd ohci_hcd uhci_hcd usb_storage ehci_hcd mpt2sas raid_class aic94xx libsas lpfc qla2xxx megaraid_sas megaraid_mbox megaraid_mm megaraid aacraid sx8 DAC960 hpsa cciss 3w_9xxx 3w_xxxx mptsas scsi_transport_sas mptfc scsi_transport_fc scsi_tgt mptspi mptscsih mptbase atp870u dc395x qla1280 dmx3191d sym53c8xx gdth advansys initio BusLogic arcmsr aic7xxx aic79xx scsi_transport_spi sg pdc_adma sata_inic162x sata_mv ata_piix ahci libahci sata_qstor sata_vsc sata_uli sata_sis sata_sx4 sata_nv sata_via sata_svw sata_sil24 sata_sil sata_promise pata_sl82c105 pata_cs5530 pata_cs5520 pata_via pata_jmicron pata_marvell pata_sis pata_netcell pata_sc1200 pata_pdc202xx_old pata_triflex pata_atiixp pata_opti pata_amd pata_ali pata_it8213 pata_ns87415 pata_ns87410 pata_serverworks pata_cypress pata_artop pata_it821x pata_optidma pata_hpt3x2n pata_hpt3x3 pata_hpt37x pata_hpt366 pata_cmd64x pata_efar pata_rz1000 pata_sil680 pata_radisys pata_pdc2027x pata_mpiix libata [last unloaded: scsi_transport_iscsi]
Aug 22 15:33:47 primary kernel: : [1489938.428324] Pid: 0, comm: swapper/0 Tainted: G W 3.8.1-gentoo #1
Aug 22 15:33:47 primary kernel: : [1489938.428326] Call Trace:
Aug 22 15:33:47 primary kernel: : [1489938.428373] <IRQ> [<ffffffff8155ba00>] ? skb_warn_bad_offload+0x10/0xc0
Aug 22 15:33:47 primary kernel: : [1489938.428385] [<ffffffff81075d30>] ? warn_slowpath_common+0x80/0xc0
Aug 22 15:33:47 primary kernel: : [1489938.428390] [<ffffffff81075e2a>] ? warn_slowpath_fmt+0x4a/0x50
Aug 22 15:33:47 primary kernel: : [1489938.428395] [<ffffffff8155baa7>] ? skb_warn_bad_offload+0xb7/0xc0
Aug 22 15:33:47 primary kernel: : [1489938.428400] [<ffffffff810a3358>] ? wake_up+0x48/0x70
Aug 22 15:33:47 primary kernel: : [1489938.428408] [<ffffffff81463666>] ? skb_gso_segment+0x186/0x250
Aug 22 15:33:47 primary kernel: : [1489938.428415] [<ffffffff81175cc8>] ? kmalloc_node_track_caller+0x38/0x210
Aug 22 15:33:47 primary kernel: : [1489938.428421] [<ffffffff8146786d>] ? dev_hard_start_xmit+0xbd/0x570
Aug 22 15:33:47 primary kernel: : [1489938.428428] [<ffffffff81482e5e>] ? sch_direct_xmit+0x10e/0x1e0
Aug 22 15:33:47 primary kernel: : [1489938.428433] [<ffffffff814680cd>] ? dev_queue_xmit+0x15d/0x450
Aug 22 15:33:47 primary kernel: : [1489938.428442] [<ffffffffa0963c98>] ? br_dev_queue_push_xmit+0x88/0xc0 [bridge]
Aug 22 15:33:47 primary kernel: : [1489938.428449] [<ffffffffa096a407>] ? br_sysfs_delbr+0x607/0x1cd0 [bridge]
Aug 22 15:33:47 primary kernel: : [1489938.428456] [<ffffffffa0963c10>] ? br_fdb_delete+0x3a0/0x3a0 [bridge]
Aug 22 15:33:47 primary kernel: : [1489938.428462] [<ffffffff8148ffe6>] ? nf_iterate+0x96/0xd0
Aug 22 15:33:47 primary kernel: : [1489938.428470] [<ffffffffa0963c10>] ? br_fdb_delete+0x3a0/0x3a0 [bridge]
Aug 22 15:33:47 primary kernel: : [1489938.428475] [<ffffffff8149009f>] ? nf_hook_slow+0x7f/0x160
Aug 22 15:33:47 primary kernel: : [1489938.428482] [<ffffffffa0963c10>] ? br_fdb_delete+0x3a0/0x3a0 [bridge]
Aug 22 15:33:47 primary kernel: : [1489938.428489] [<ffffffffa0963cd0>] ? br_dev_queue_push_xmit+0xc0/0xc0 [bridge]
Aug 22 15:33:47 primary kernel: : [1489938.428496] [<ffffffffa0963d1a>] ? br_forward_finish+0x4a/0x170 [bridge]
Aug 22 15:33:47 primary kernel: : [1489938.428503] [<ffffffffa096a554>] ? br_sysfs_delbr+0x754/0x1cd0 [bridge]
Aug 22 15:33:47 primary kernel: : [1489938.428510] [<ffffffffa096b138>] ? br_sysfs_delbr+0x1338/0x1cd0 [bridge]
Aug 22 15:33:47 primary kernel: : [1489938.428515] [<ffffffff8148ffe6>] ? nf_iterate+0x96/0xd0
Aug 22 15:33:47 primary kernel: : [1489938.428522] [<ffffffffa0963cd0>] ? br_dev_queue_push_xmit+0xc0/0xc0 [bridge]
Aug 22 15:33:47 primary kernel: : [1489938.428527] [<ffffffff8149009f>] ? nf_hook_slow+0x7f/0x160
Aug 22 15:33:47 primary kernel: : [1489938.428533] [<ffffffffa0963cd0>] ? br_dev_queue_push_xmit+0xc0/0xc0 [bridge]
Aug 22 15:33:47 primary kernel: : [1489938.428540] [<ffffffffa0963d30>] ? br_forward_finish+0x60/0x170 [bridge]
Aug 22 15:33:47 primary kernel: : [1489938.428547] [<ffffffffa09649b0>] ? br_net_exit+0xe0/0xe0 [bridge]
Aug 22 15:33:47 primary kernel: : [1489938.428554] [<ffffffffa0963db0>] ? br_forward_finish+0xe0/0x170 [bridge]
Aug 22 15:33:47 primary kernel: : [1489938.428561] [<ffffffffa096399b>] ? br_fdb_delete+0x12b/0x3a0 [bridge]
Aug 22 15:33:47 primary kernel: : [1489938.428568] [<ffffffffa0964b3f>] ? br_handle_frame_finish+0x18f/0x2c0 [bridge]
Aug 22 15:33:47 primary kernel: : [1489938.428575] [<ffffffffa096aab0>] ? br_sysfs_delbr+0xcb0/0x1cd0 [bridge]
Aug 22 15:33:47 primary kernel: : [1489938.428582] [<ffffffffa096ac6b>] ? br_sysfs_delbr+0xe6b/0x1cd0 [bridge]
Aug 22 15:33:47 primary kernel: : [1489938.428589] [<ffffffffa096b7bb>] ? br_sysfs_delbr+0x19bb/0x1cd0 [bridge]
Aug 22 15:33:47 primary kernel: : [1489938.428594] [<ffffffff8148ffe6>] ? nf_iterate+0x96/0xd0
Aug 22 15:33:47 primary kernel: : [1489938.428602] [<ffffffffa09649b0>] ? br_net_exit+0xe0/0xe0 [bridge]
Aug 22 15:33:47 primary kernel: : [1489938.428607] [<ffffffff8149009f>] ? nf_hook_slow+0x7f/0x160
Aug 22 15:33:47 primary kernel: : [1489938.428614] [<ffffffffa09649b0>] ? br_net_exit+0xe0/0xe0 [bridge]
Aug 22 15:33:47 primary kernel: : [1489938.428621] [<ffffffffa0964e50>] ? br_handle_frame+0x1e0/0x9e0 [bridge]
Aug 22 15:33:47 primary kernel: : [1489938.428628] [<ffffffffa0964c70>] ? br_handle_frame_finish+0x2c0/0x2c0 [bridge]
Aug 22 15:33:47 primary kernel: : [1489938.428634] [<ffffffff81465027>] ? netif_receive_skb+0x167/0x780
Aug 22 15:33:47 primary kernel: : [1489938.428640] [<ffffffff814657bf>] ? netif_receive_skb+0x1f/0x80
Aug 22 15:33:47 primary kernel: : [1489938.428645] [<ffffffff81466838>] ? napi_gro_receive+0xd8/0x120
Aug 22 15:33:47 primary kernel: : [1489938.428654] [<ffffffffa08d81c5>] ? bnx2x_rx_int+0xb35/0x1480 [bnx2x]
Aug 22 15:33:47 primary kernel: : [1489938.428661] [<ffffffff810b3257>] ? load_balance+0x107/0x7d0
Aug 22 15:33:47 primary kernel: : [1489938.428711] [<ffffffffa08d8bba>] ? bnx2x_poll+0xaa/0x2e0 [bnx2x]
Aug 22 15:33:47 primary kernel: : [1489938.428716] [<ffffffff81465a4d>] ? net_rx_action+0x8d/0x1a0
Aug 22 15:33:47 primary kernel: : [1489938.428723] [<ffffffff8107de2a>] ? do_softirq+0xba/0x240
Aug 22 15:33:47 primary kernel: : [1489938.428730] [<ffffffff815608cc>] ? call_softirq+0x1c/0x30
Aug 22 15:33:47 primary kernel: : [1489938.428776] [<ffffffff8103ebd5>] ? do_softirq+0x55/0x90
Aug 22 15:33:47 primary kernel: : [1489938.428782] [<ffffffff8107e0ee>] ? irq_exit+0x8e/0xb0
Aug 22 15:33:47 primary kernel: : [1489938.428786] [<ffffffff81560f61>] ? do_IRQ+0x61/0xe0
Aug 22 15:33:47 primary kernel: : [1489938.428791] [<ffffffff8155ed6a>] ? common_interrupt+0x6a/0x6a
Aug 22 15:33:47 primary kernel: : [1489938.428793] <EOI> [<ffffffff8142710e>] ? cpuidle_wrap_enter+0x4e/0xa0
Aug 22 15:33:47 primary kernel: : [1489938.428804] [<ffffffff8142710a>] ? cpuidle_wrap_enter+0x4a/0xa0
Aug 22 15:33:47 primary kernel: : [1489938.428809] [<ffffffff81426d29>] ? cpuidle_idle_call+0xa9/0x290
Aug 22 15:33:47 primary kernel: : [1489938.428815] [<ffffffff810464ef>] ? cpu_idle+0x7f/0xd0
Aug 22 15:33:47 primary kernel: : [1489938.428820] [<ffffffff818e6b3c>] ? start_kernel+0x365/0x370
Aug 22 15:33:47 primary kernel: : [1489938.428824] [<ffffffff818e65f0>] ? repair_env_string+0x58/0x58
Aug 22 15:33:47 primary kernel: : [1489938.428828] ---[ end trace 16a741dd8b9b00d9 ]---

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Stefan Mühling - 2013-08-27

Happend at another costumer after changing NICs from 1Gbit (onboard) to 10Gbit (Intel X540 T2).

[67003.135797] ------------[ cut here ]------------
[67003.135808] WARNING: at net/core/dev.c:2029 skb_warn_bad_offload+0xb7/0xc0()
[67003.135810] Hardware name: ProLiant DL360 G7
[67003.135814] : caps=(0x0000000040004849, 0x0000000000000000) len=1440 data_len=1420 gso_size=1420 gso_type=1 ip_summed=1
[67003.135815] Modules linked in: vhost_net macvtap macvlan gfs2 dlm sctp configfs ebtable_nat ebtables bridge stp llc bonding coretemp mperf freq_table crc32c_intel ghash_clmulni_intel ixgbe ehci_pci i7core_edac microcode iTCO_wdt edac_core mdio iTCO_vendor_support lpc_ich hpilo mfd_core hpwdt bnx2 acpi_power_meter kvm_intel button aesni_intel ablk_helper cryptd lrw xts gf128mul aes_x86_64 sha256_generic iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi tg3 libphy hwmon ptp pps_core e1000 fuse nfs fscache lockd sunrpc reiserfs btrfs zlib_deflate lzo_compress ext4 jbd2 ext3 jbd dm_crypt dm_mirror dm_region_hash dm_log sl811_hcd xhci_hcd ohci_hcd uhci_hcd usb_storage ehci_hcd mpt2sas raid_class aic94xx libsas lpfc qla2xxx megaraid_sas megaraid_mbox megaraid_mm megaraid aacraid sx8 DAC960 hpsa cciss
[67003.135888] 3w_9xxx 3w_xxxx mptsas scsi_transport_sas mptfc scsi_transport_fc scsi_tgt mptspi mptscsih mptbase atp870u dc395x qla1280 dmx3191d sym53c8xx gdth advansys initio BusLogic arcmsr aic7xxx aic79xx scsi_transport_spi sg pdc_adma sata_inic162x sata_mv ata_piix ahci libahci sata_qstor sata_vsc sata_uli sata_sis sata_sx4 sata_nv sata_via sata_svw sata_sil24 sata_sil sata_promise pata_sl82c105 pata_cs5530 pata_cs5520 pata_via pata_jmicron pata_marvell pata_sis pata_netcell pata_sc1200 pata_pdc202xx_old pata_triflex pata_atiixp pata_opti pata_amd pata_ali pata_it8213 pata_ns87415 pata_ns87410 pata_serverworks pata_cypress pata_artop pata_it821x pata_optidma pata_hpt3x2n pata_hpt3x3 pata_hpt37x pata_hpt366 pata_cmd64x pata_efar pata_rz1000 pata_sil680 pata_radisys pata_pdc2027x pata_mpiix libata

[67003.135950] Pid: 0, comm: swapper/0 Tainted: G W 3.8.1-gentoo #1
[67003.135952] Call Trace:
[67003.135953] <IRQ> [<ffffffff8155ba00>] ? skb_warn_bad_offload+0x10/0xc0
[67003.135962] [<ffffffff81075d30>] ? warn_slowpath_common+0x80/0xc0
[67003.135965] [<ffffffff81075e2a>] ? warn_slowpath_fmt+0x4a/0x50
[67003.135970] [<ffffffffa0a37b1c>] ? vhost_work_queue+0x7c/0x90 [vhost_net]
[67003.135974] [<ffffffff8155baa7>] ? skb_warn_bad_offload+0xb7/0xc0
[67003.135979] [<ffffffff81463666>] ? skb_gso_segment+0x186/0x250
[67003.135984] [<ffffffff81175cc8>] ? kmalloc_node_track_caller+0x38/0x210
[67003.135988] [<ffffffff8146786d>] ? dev_hard_start_xmit+0xbd/0x570
[67003.135993] [<ffffffff81482e5e>] ? sch_direct_xmit+0x10e/0x1e0
[67003.135997] [<ffffffff814680cd>] ? dev_queue_xmit+0x15d/0x450
[67003.136002] [<ffffffffa0916c98>] ? br_dev_queue_push_xmit+0x88/0xc0 [bridge]
[67003.136007] [<ffffffffa091d407>] ? br_sysfs_delbr+0x607/0x1cd0 [bridge]
[67003.136012] [<ffffffffa0916c10>] ? br_fdb_delete+0x3a0/0x3a0 [bridge]
[67003.136016] [<ffffffff8148ffe6>] ? nf_iterate+0x96/0xd0
[67003.136021] [<ffffffffa0916c10>] ? br_fdb_delete+0x3a0/0x3a0 [bridge]
[67003.136024] [<ffffffff8149009f>] ? nf_hook_slow+0x7f/0x160
[67003.136028] [<ffffffffa0916c10>] ? br_fdb_delete+0x3a0/0x3a0 [bridge]
[67003.136032] [<ffffffffa0916cd0>] ? br_dev_queue_push_xmit+0xc0/0xc0 [bridge]
[67003.136037] [<ffffffffa0916d1a>] ? br_forward_finish+0x4a/0x170 [bridge]
[67003.136041] [<ffffffffa091d554>] ? br_sysfs_delbr+0x754/0x1cd0 [bridge]
[67003.136045] [<ffffffffa091e138>] ? br_sysfs_delbr+0x1338/0x1cd0 [bridge]
[67003.136049] [<ffffffff8148ffe6>] ? nf_iterate+0x96/0xd0
[67003.136053] [<ffffffffa0916cd0>] ? br_dev_queue_push_xmit+0xc0/0xc0 [bridge]
[67003.136057] [<ffffffff8149009f>] ? nf_hook_slow+0x7f/0x160
[67003.136061] [<ffffffffa0916cd0>] ? br_dev_queue_push_xmit+0xc0/0xc0 [bridge]
[67003.136065] [<ffffffffa0916d30>] ? br_forward_finish+0x60/0x170 [bridge]
[67003.136070] [<ffffffffa09179b0>] ? br_net_exit+0xe0/0xe0 [bridge]
[67003.136074] [<ffffffffa0916db0>] ? br_forward_finish+0xe0/0x170 [bridge]
[67003.136078] [<ffffffffa091699b>] ? br_fdb_delete+0x12b/0x3a0 [bridge]
[67003.136083] [<ffffffffa0917b3f>] ? br_handle_frame_finish+0x18f/0x2c0 [bridge]
[67003.136087] [<ffffffffa091dab0>] ? br_sysfs_delbr+0xcb0/0x1cd0 [bridge]
[67003.136091] [<ffffffffa091dc6b>] ? br_sysfs_delbr+0xe6b/0x1cd0 [bridge]
[67003.136096] [<ffffffffa091e7bb>] ? br_sysfs_delbr+0x19bb/0x1cd0 [bridge]
[67003.136099] [<ffffffff8148ffe6>] ? nf_iterate+0x96/0xd0
[67003.136104] [<ffffffffa09179b0>] ? br_net_exit+0xe0/0xe0 [bridge]
[67003.136107] [<ffffffff8149009f>] ? nf_hook_slow+0x7f/0x160
[67003.136112] [<ffffffffa09179b0>] ? br_net_exit+0xe0/0xe0 [bridge]
[67003.136117] [<ffffffffa0917e50>] ? br_handle_frame+0x1e0/0x9e0 [bridge]
[67003.136121] [<ffffffffa0917c70>] ? br_handle_frame_finish+0x2c0/0x2c0 [bridge]
[67003.136125] [<ffffffff81465027>] ? netif_receive_skb+0x167/0x780
[67003.136128] [<ffffffff814657bf>] ? netif_receive_skb+0x1f/0x80
[67003.136132] [<ffffffff81466838>] ? napi_gro_receive+0xd8/0x120
[67003.136137] [<ffffffffa0aae38c>] ? ixgbe_poll+0x56c/0x1300 [ixgbe]
[67003.136141] [<ffffffff81092100>] ? queue_work+0x3f0/0x3f0
[67003.136145] [<ffffffff81465a4d>] ? net_rx_action+0x8d/0x1a0
[67003.136149] [<ffffffff810b3ada>] ? run_rebalance_domains+0x4a/0x190
[67003.136154] [<ffffffff8107de2a>] ? do_softirq+0xba/0x240
[67003.136160] [<ffffffff815608cc>] ? call_softirq+0x1c/0x30
[67003.136164] [<ffffffff8103ebd5>] ? do_softirq+0x55/0x90
[67003.136167] [<ffffffff8107e0ee>] ? irq_exit+0x8e/0xb0
[67003.136171] [<ffffffff81560f61>] ? do_IRQ+0x61/0xe0
[67003.136174] [<ffffffff8155ed6a>] ? common_interrupt+0x6a/0x6a
[67003.136175] <EOI> [<ffffffff8109d373>] ? __hrtimer_start_range_ns+0x1e3/0x490
[67003.136184] [<ffffffff8142710e>] ? cpuidle_wrap_enter+0x4e/0xa0
[67003.136187] [<ffffffff8142710a>] ? cpuidle_wrap_enter+0x4a/0xa0
[67003.136190] [<ffffffff81426d29>] ? cpuidle_idle_call+0xa9/0x290
[67003.136194] [<ffffffff810464ef>] ? cpu_idle+0x7f/0xd0
[67003.136197] [<ffffffff818e6b3c>] ? start_kernel+0x365/0x370
[67003.136200] [<ffffffff818e65f0>] ? repair_env_string+0x58/0x58
[67003.136202] ---[ end trace 76e506f939dfb97e ]---

There are several threads concerning skb_warn_bad_offload in conjunction with bridging.

I tried different drivers inside the virtual machines (ranging from 0.49 to 0.65). Operating system is Win2K8 R2.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Problem with NIC

Build your own Cloud

Group

Searches

Help

#270 Problem with NIC

Discussion