#3 82541GI Tx Unit Hang

wont-fix
e1000 (137)
standalone_driver
5
2014-09-03
2011-06-21
Anonymous
No

On our production load-balancers we routinely see Tx Unit Hangs. Both are identical spec servers, running CentOS, and using corosync for failover.
I've got a third identical server sitting in our test-environment which never sees this happen though admittedly it's not under the same level of traffic (typically around 6-8Mb/sec of steady traffic)
Using the driver included with the CentOS kernel (7.something) has it being really unstable, so generally I've been using ones from here. After upgrading the kernel recently I let it use stock for a while but that wasn't stable, so grabbed the very latest from here (8.0.30). Whilst it's a huge improvement we do still see failures:

Jun 20 18:17:42 server-name kernel: e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
Jun 20 18:17:42 server-name kernel: Tx Queue <0>
Jun 20 18:17:42 server-name kernel: TDH <4d>
Jun 20 18:17:42 server-name kernel: TDT <ae>
Jun 20 18:17:42 server-name kernel: next_to_use <ae>
Jun 20 18:17:42 server-name kernel: next_to_clean <47>
Jun 20 18:17:42 server-name kernel: buffer_info[next_to_clean]
Jun 20 18:17:42 server-name kernel: time_stamp <10f22237a>
Jun 20 18:17:42 server-name kernel: next_to_watch <4e>
Jun 20 18:17:42 server-name kernel: jiffies <10f223335>
Jun 20 18:17:42 server-name kernel: next_to_watch.status <0>
Jun 20 18:17:44 server-name kernel: e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
Jun 20 18:17:49 server-name kernel: Tx Queue <0>
Jun 20 18:17:49 server-name kernel: TDH <4d>
Jun 20 18:17:49 server-name kernel: TDT <ae>
Jun 20 18:17:49 server-name kernel: next_to_use <ae>
Jun 20 18:17:49 server-name kernel: next_to_clean <47>
Jun 20 18:17:49 server-name kernel: buffer_info[next_to_clean]
Jun 20 18:17:49 server-name kernel: time_stamp <10f22237a>
Jun 20 18:17:49 server-name kernel: next_to_watch <4e>
Jun 20 18:17:49 server-name kernel: jiffies <10f223b05>
Jun 20 18:17:49 server-name kernel: next_to_watch.status <0>
Jun 20 18:17:49 server-name kernel: e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
Jun 20 18:17:49 server-name kernel: Tx Queue <0>
Jun 20 18:17:49 server-name kernel: TDH <4d>
Jun 20 18:17:49 server-name kernel: TDT <ae>
Jun 20 18:17:49 server-name kernel: next_to_use <ae>
Jun 20 18:17:49 server-name kernel: next_to_clean <47>
Jun 20 18:17:49 server-name kernel: buffer_info[next_to_clean]
Jun 20 18:17:49 server-name kernel: time_stamp <10f22237a>
Jun 20 18:17:49 server-name kernel: next_to_watch <4e>
Jun 20 18:17:49 server-name kernel: jiffies <10f2242d5>
Jun 20 18:17:49 server-name kernel: next_to_watch.status <0>
Jun 20 18:17:49 server-name kernel: e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
Jun 20 18:17:49 server-name kernel: Tx Queue <0>
Jun 20 18:17:49 server-name kernel: TDH <4d>
Jun 20 18:17:49 server-name kernel: TDT <ae>
Jun 20 18:17:49 server-name kernel: next_to_use <ae>
Jun 20 18:17:49 server-name kernel: next_to_clean <47>
Jun 20 18:17:49 server-name kernel: buffer_info[next_to_clean]
Jun 20 18:17:49 server-name kernel: time_stamp <10f22237a>
Jun 20 18:17:49 server-name kernel: next_to_watch <4e>
Jun 20 18:17:49 server-name kernel: jiffies <10f224aa6>
Jun 20 18:17:49 server-name kernel: next_to_watch.status <0>
Jun 20 18:17:49 server-name kernel: NETDEV WATCHDOG: eth0: transmit timed out
Jun 20 18:17:51 server-name kernel: e1000: eth0: e1000_watchdog_task: NIC Link is Up 100 Mbps Full Duplex, Flow Control: None

*-network
description: Ethernet interface
product: 82541GI Gigabit Ethernet Controller
vendor: Intel Corporation
physical id: 4
bus info: pci@0000:02:04.0
logical name: eth0
version: 05
serial: 00:15:c5:5d:2d:8e
size: 100MB/s
capacity: 1GB/s
width: 32 bits
clock: 66MHz
capabilities: pm pcix bus_master cap_list ethernet physical tp 10bt 10bt-fd 100bt 100bt-fd 1000bt-fd autonegotiation
configuration: autonegotiation=on broadcast=yes driver=e1000 driverversion=8.0.30-NAPI duplex=full firmware=N/A ip=??.??.??.?? latency=32 link=yes mingnt=255 module=e1000 multicast=yes port=twisted pair speed=100MB/s

I've removed a few bits of sensitive data. Unfortunately I'm unable to re-create the problem, even if I push similar traffic levels through our test server. I'm wondering if multicast packets, or certain unusually shaped packets, might be tied into the problem. Attempts to automatically catch the exception and run a tcpdump have so far failed to succeed on capturing a repeat occurrence, but I'll keep trying on that. It's frequent enough to be annoying, but not frequent enough to enable me to just leave tcpdump running. In the shorter term we'll be replacing the servers, but thought I'd add this ticket in case there is anything useful from what we're seeing.

Discussion

1 2 3 > >> (Page 1 of 3)
  • Tushar Dave

    Tushar Dave - 2011-06-21

    Sorry that you have this issue.
    Please try loading the driver with module parameter ignore_64bit_dma=1
    This is a load time parameter only so first unload the driver and then load driver with module parameter. (i.e rmmod e1000; insmod ./e1000.ko ignore_64bit_dma=1)
    Once driver is loaded you can also confirm that module parameter is set via

    cat /sys/module/e1000/parameters/ignore_64bit_dma , it should return 1.

     
  • Comment has been marked as spam. 
    Undo

    You can see all pending comments posted by this user  here

    Anonymous - 2011-06-22

    I added the ignore_64bit_dma flag on the servers last night and will leave it for a couple of days to see how things go.

    Strangely I'd got that set in modprobe.conf (see below) but it seems to have ignored the setting, based on the output from cat. I'll have to figure that quirk out on our test-box...

    alias eth0 e1000 ignore_64bit_dma=1 XsumRX=0 InterruptThrottleRate=0 FlowControl=3 RxDescriptors=4096 TxDescriptors=4096 RxIntDelay=0 TxIntDelay=0

     
  • Comment has been marked as spam. 
    Undo

    You can see all pending comments posted by this user  here

    Anonymous - 2011-06-23

    Sadly that hasn't cleared it, four of these cropped up:

    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <13>
    TDT <99>
    next_to_use <99>
    next_to_clean <d>
    buffer_info[next_to_clean]
    time_stamp <118f0d37a>
    next_to_watch <16>
    jiffies <118f0f33b>
    next_to_watch.status <0>

     
  • Jesse Brandeburg

    your syntax is incorrect in modprobe files (see man modprobe.conf).

    should be
    alias eth0 e1000
    options e1000 ignore_64bit_dma=1 XsumRX=0 InterruptThrottleRate=0
    FlowControl=3 RxDescriptors=4096 TxDescriptors=4096 RxIntDelay=0
    TxIntDelay=0

    see Tushar's instructions for verifying your module parameter for ignore_64_bit_dma is set.

    setting RxIntDelay and TxIntDelay may introduce bugs, FYI, and we would prefer for debugging that you didn't set anything but InterruptThrottleRate and RxDescriptors

    why do you disable XsumRX? All that is going to cost you is lots of cpu with nothing to gain. I'm curious for your reasoning here or if you are just shotgunning options.

     
  • Comment has been marked as spam. 
    Undo

    You can see all pending comments posted by this user  here

    Anonymous - 2011-06-26

    Sadly still seeing the problem.

    cat /sys/module/e1000/parameters/ignore_64bit_dma

    1
    On both of them :-/ Wish I could track down what kind of packet it is that's coming through and causing this so I could see if I could filter it!

     
  • Comment has been marked as spam. 
    Undo

    You can see all pending comments posted by this user  here

    Anonymous - 2011-06-26

    Should clarify by my last update, I meant that I'm still getting the Tx Unit Hang, on both, and that /sys/module/e1000/parameters/ignore_64bit_dma is returning 1 on both of them :D

     
  • Tushar Dave

    Tushar Dave - 2011-06-27

    Did you see comment from go_Jesse. For debugging we would prefer that you don't set any other module parameter except InterruptThrottleRateand RxDescriptors.

    How is your modprobe.conf look like for e1000 driver?

     
  • Comment has been marked as spam. 
    Undo

    You can see all pending comments posted by this user  here

    Anonymous - 2011-06-30

    I've unloaded and reloaded the module with the following settings:

    alias eth0 e1000
    alias eth1 e1000
    options e1000 ignore_64bit_dma=1 InterruptThrottleRate=0 RxDescriptors=4096

    The other entries were in there from a while back when I first investigated the problem, seemed to be the advice at the time and what other people were finding worked for them.

     
  • Comment has been marked as spam. 
    Undo

    You can see all pending comments posted by this user  here

    Anonymous - 2011-06-30

    Unfortunately, problem remains :(

     
  • Tushar Dave

    Tushar Dave - 2011-06-30

    Sorry to hear that the issue remains.
    My idea was to first use the known workaround before diving into details.

    In order to investigate this issue further, please send the output of
    uname -a
    cat /proc/interrupts
    ethtool -i ethx
    ethtool -S ethx
    full dmesg log

    I can also provide you a debug driver which dump HW ring when Tx hang occurs. It logs all info into dmesg log and won't cause any significant performance impact on system. Would you be interested in trying debug driver?

    One more thing FYI, in my recent experience I have seen 82541 got hung while accessing data from certain (range of) DMA addresses that resulted in PCI Master Abort bus error. Our debugging found that there is nothing that driver was doing wrong. DMA address assigned to driver from kernel and certainly it turned out to be not a driver or HW issue but kernel. Upgrading the kernel with latest version 3.0.0 has resolved the issue. In this regards, would you be willing to try latest kernel?

     
  • Comment has been marked as spam. 
    Undo

    You can see all pending comments posted by this user  here

    Anonymous - 2011-06-30

    Linux ** 2.6.18-238.12.1.el5 #1 SMP Tue May 31 13:22:04 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux

    cat /proc/interrupts

           CPU0       CPU1       CPU2       CPU3       CPU4       CPU5       CPU6       CPU7
    

    0: 587374579 0 0 1 0 99 4 56252 IO-APIC-edge timer
    1: 0 0 0 0 0 1 0 2 IO-APIC-edge i8042
    8: 0 0 0 0 0 0 0 1 IO-APIC-edge rtc
    9: 0 0 0 0 0 2 0 31 IO-APIC-level acpi
    10: 0 0 1146237 45301 93486 786284 229550 9151 IO-APIC-level sata_svw
    11: 0 0 0 0 11 0 0 54 IO-APIC-level ehci_hcd:usb1, ohci_hcd:usb2, ohci_hcd:usb3
    12: 0 0 0 0 0 0 0 5 IO-APIC-edge i8042
    66: 255 0 0 0 8903 0 11038220 49059820 PCI-MSI eth0
    74: 41 0 0 0 0 0 49595 10272782 PCI-MSI eth1
    NMI: 28262 24253 24888 25213 24401 23829 25853 28634
    LOC: 587423368 587423318 587423235 587423168 587423096 587423023 587422910 587422845
    ERR: 0
    MIS: 0

    ethtool -i eth0
    driver: tg3
    version: 3.108
    firmware-version: 5721-v3.61, ASFIPMI v6.21
    bus-info: 0000:01:00.0

    ethtool -S eth0

    NIC statistics:
    rx_octets: 58550900416
    rx_fragments: 0
    rx_ucast_packets: 44746456
    rx_mcast_packets: 75
    rx_bcast_packets: 374560
    rx_fcs_errors: 0
    rx_align_errors: 0
    rx_xon_pause_rcvd: 0
    rx_xoff_pause_rcvd: 0
    rx_mac_ctrl_rcvd: 0
    rx_xoff_entered: 0
    rx_frame_too_long_errors: 0
    rx_jabbers: 0
    rx_undersize_packets: 0
    rx_in_length_errors: 0
    rx_out_length_errors: 0
    rx_64_or_less_octet_packets: 0
    rx_65_to_127_octet_packets: 0
    rx_128_to_255_octet_packets: 0
    rx_256_to_511_octet_packets: 0
    rx_512_to_1023_octet_packets: 0
    rx_1024_to_1522_octet_packets: 0
    rx_1523_to_2047_octet_packets: 0
    rx_2048_to_4095_octet_packets: 0
    rx_4096_to_8191_octet_packets: 0
    rx_8192_to_9022_octet_packets: 0
    tx_octets: 2039915520
    tx_collisions: 0
    tx_xon_sent: 0
    tx_xoff_sent: 0
    tx_flow_control: 0
    tx_mac_errors: 0
    tx_single_collisions: 0
    tx_mult_collisions: 0
    tx_deferred: 0
    tx_excessive_collisions: 0
    tx_late_collisions: 0
    tx_collide_2times: 0
    tx_collide_3times: 0
    tx_collide_4times: 0
    tx_collide_5times: 0
    tx_collide_6times: 0
    tx_collide_7times: 0
    tx_collide_8times: 0
    tx_collide_9times: 0
    tx_collide_10times: 0
    tx_collide_11times: 0
    tx_collide_12times: 0
    tx_collide_13times: 0
    tx_collide_14times: 0
    tx_collide_15times: 0
    tx_ucast_packets: 19577485
    tx_mcast_packets: 6
    tx_bcast_packets: 458
    tx_carrier_sense_errors: 0
    tx_discards: 0
    tx_errors: 0
    dma_writeq_full: 0
    dma_write_prioq_full: 0
    rxbds_empty: 0
    rx_discards: 0
    rx_errors: 0
    rx_threshold_hit: 0
    dma_readq_full: 0
    dma_read_prioq_full: 0
    tx_comp_queue_full: 0
    ring_set_send_prod_index: 0
    ring_status_update: 0
    nic_irqs: 0
    nic_avoided_irqs: 0
    nic_tx_threshold_hit: 0

    Linux version 2.6.18-238.12.1.el5 (mockbuild@builder10.centos.org) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-50)) #1 SMP Tue May 31 13:22:04 EDT 2011
    Command line: ro root=/dev/VolGroup00/LogVol00
    BIOS-provided physical RAM map:
    BIOS-e820: 0000000000010000 - 00000000000a0000 (usable)
    BIOS-e820: 0000000000100000 - 00000000dfed2000 (usable)
    BIOS-e820: 00000000dfed2000 - 00000000dfee8000 (reserved)
    BIOS-e820: 00000000dfee8000 - 00000000dff07c00 (ACPI data)
    BIOS-e820: 00000000dff07c00 - 00000000e0000000 (reserved)
    BIOS-e820: 00000000f0000000 - 00000000f8000000 (reserved)
    BIOS-e820: 00000000fe000000 - 0000000100000000 (reserved)
    BIOS-e820: 0000000100000000 - 0000000220000000 (usable)
    DMI 2.4 present.
    ACPI: RSDP (v002 DELL ) @ 0x00000000000f1e50
    ACPI: XSDT (v001 DELL PE_SC3 0x00000001 DELL 0x00000001) @ 0x00000000000f1ec8
    ACPI: FADT (v003 DELL PE_SC3 0x00000001 DELL 0x00000001) @ 0x00000000000f1fc0
    ACPI: MADT (v001 DELL PE_SC3 0x00000001 DELL 0x00000001) @ 0x00000000000f20b4
    ACPI: SPCR (v001 DELL PE_SC3 0x00000001 DELL 0x00000001) @ 0x00000000000f2155
    ACPI: HPET (v001 DELL PE_SC3 0x00000001 DELL 0x00000001) @ 0x00000000000f21a5
    ACPI: MCFG (v001 DELL PE_SC3 0x00000001 DELL 0x00000001) @ 0x00000000000f21dd
    ACPI: SLIC (v001 DELL PE_SC3 0x00000001 DELL 0x00000001) @ 0x00000000000f2219
    ACPI: ERST (v001 DELL PE_SC3 0x00000001 DELL 0x00000001) @ 0x00000000dfeebf48
    ACPI: HEST (v001 DELL PE_SC3 0x00000001 DELL 0x00000001) @ 0x00000000dfeec158
    ACPI: BERT (v001 DELL PE_SC3 0x00000001 DELL 0x00000001) @ 0x00000000dfeebdc8
    ACPI: EINJ (v001 DELL PE_SC3 0x00000001 DELL 0x00000001) @ 0x00000000dfeebdf8
    ACPI: SRAT (v001 DELL PE_SC3 0x00000001 DELL 0x00000001) @ 0x00000000000fc084
    ACPI: SSDT (v001 DELL PE_SC3 0x00000001 DELL 0x00000001) @ 0x00000000dff05000
    ACPI: DSDT (v001 DELL PE_SC3 0x00000001 INTL 0x20050624) @ 0x0000000000000000
    SRAT: PXM 0 -> APIC 0 -> Node 0
    SRAT: PXM 0 -> APIC 1 -> Node 0
    SRAT: PXM 0 -> APIC 2 -> Node 0
    SRAT: PXM 0 -> APIC 3 -> Node 0
    SRAT: PXM 1 -> APIC 4 -> Node 1
    SRAT: PXM 1 -> APIC 5 -> Node 1
    SRAT: PXM 1 -> APIC 6 -> Node 1
    SRAT: PXM 1 -> APIC 7 -> Node 1
    SRAT: Node 0 PXM 0 0-a0000
    SRAT: Node 0 PXM 0 0-e0000000
    SRAT: Node 0 PXM 0 0-120000000
    SRAT: Node 1 PXM 1 120000000-220000000
    NUMA: Using 29 for the hash shift.
    Bootmem setup node 0 0000000000000000-0000000120000000
    bootmap start 20000 pages 36
    Bootmem setup node 1 0000000120000000-0000000220000000
    bootmap start 120006000 pages 32
    Memory for crash kernel (0x0 to 0x0) notwithin permissible range
    disabling kdump
    Setting up node 0 0-120000
    On node 0 totalpages: 1030540
    DMA zone: 2434 pages, LIFO batch:0
    DMA32 zone: 898826 pages, LIFO batch:31
    Normal zone: 129280 pages, LIFO batch:31
    Setting up node 1 120000-220000
    On node 1 totalpages: 1034240
    Normal zone: 1034240 pages, LIFO batch:31
    ACPI: PM-Timer IO Port: 0x808
    ACPI: Local APIC address 0xfee00000
    ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
    Processor #0 0:4 APIC version 16
    ACPI: LAPIC (acpi_id[0x02] lapic_id[0x04] enabled)
    Processor #4 0:4 APIC version 16
    ACPI: LAPIC (acpi_id[0x03] lapic_id[0x01] enabled)
    Processor #1 0:4 APIC version 16
    ACPI: LAPIC (acpi_id[0x04] lapic_id[0x05] enabled)
    Processor #5 0:4 APIC version 16
    ACPI: LAPIC (acpi_id[0x05] lapic_id[0x02] enabled)
    Processor #2 0:4 APIC version 16
    ACPI: LAPIC (acpi_id[0x06] lapic_id[0x06] enabled)
    Processor #6 0:4 APIC version 16
    ACPI: LAPIC (acpi_id[0x07] lapic_id[0x03] enabled)
    Processor #3 0:4 APIC version 16
    ACPI: LAPIC (acpi_id[0x08] lapic_id[0x07] enabled)
    Processor #7 0:4 APIC version 16
    ACPI: LAPIC_NMI (acpi_id[0xff] high edge lint[0x1])
    ACPI: IOAPIC (id[0x08] address[0xfec00000] gsi_base[0])
    IOAPIC[0]: apic_id 8, version 17, address 0xfec00000, GSI 0-15
    ACPI: IOAPIC (id[0x09] address[0xfec01000] gsi_base[32])
    IOAPIC[1]: apic_id 9, version 17, address 0xfec01000, GSI 32-47
    ACPI: IOAPIC (id[0x0a] address[0xfec02000] gsi_base[64])
    IOAPIC[2]: apic_id 10, version 17, address 0xfec02000, GSI 64-79
    ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
    ACPI: IRQ0 used by override.
    ACPI: IRQ2 used by override.
    ACPI: IRQ9 used by override.
    Setting APIC routing to flat
    ACPI: HPET id: 0x1166a201 base: 0xfed00000
    Using ACPI (MADT) for SMP configuration information
    Nosave address range: 00000000000a0000 - 0000000000100000
    Nosave address range: 00000000dfed2000 - 00000000dfee8000
    Nosave address range: 00000000dfee8000 - 00000000dff07000
    Nosave address range: 00000000dff07000 - 00000000dff08000
    Nosave address range: 00000000dff08000 - 00000000e0000000
    Nosave address range: 00000000e0000000 - 00000000f0000000
    Nosave address range: 00000000f0000000 - 00000000f8000000
    Nosave address range: 00000000f8000000 - 00000000fe000000
    Nosave address range: 00000000fe000000 - 0000000100000000
    Allocating PCI resources starting at e1000000 (gap: e0000000:10000000)
    SMP: Allowing 8 CPUs, 0 hotplug CPUs
    Built 2 zonelists. Total pages: 2064780
    Kernel command line: ro root=/dev/VolGroup00/LogVol00
    Initializing CPU#0
    PID hash table entries: 4096 (order: 12, 32768 bytes)
    Console: colour VGA+ 80x25
    Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes)
    Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes)
    Checking aperture...
    CPU 0: aperture @ f4000000 size 64 MB
    CPU 1: aperture @ f4000000 size 64 MB
    ACPI: DMAR not present
    Memory: 8242640k/8912896k available (2592k kernel code, 144312k reserved, 1653k data, 224k init)
    Calibrating delay loop (skipped), value calculated using timer frequency.. 4588.49 BogoMIPS (lpj=2294248)
    Security Framework v1.0.0 initialized
    SELinux: Initializing.
    SELinux: Starting in permissive mode
    selinux_register_security: Registering secondary module capability
    Capability LSM initialized as secondary
    Mount-cache hash table entries: 256
    CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
    CPU: L2 Cache: 512K (64 bytes/line)
    CPU 0/0 -> Node 0
    CPU: Physical Processor ID: 0
    CPU: Processor Core ID: 0
    SMP alternatives: switching to UP code
    ACPI: Core revision 20060707
    Using local APIC timer interrupts.
    Detected 12.468 MHz APIC timer.
    SMP alternatives: switching to SMP code
    Booting processor 1/8 APIC 0x4
    Initializing CPU#1
    Calibrating delay using timer specific routine.. 4588.48 BogoMIPS (lpj=2294243)
    CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
    CPU: L2 Cache: 512K (64 bytes/line)
    CPU 1/4 -> Node 1
    CPU: Physical Processor ID: 1
    CPU: Processor Core ID: 0
    Quad-Core AMD Opteron(tm) Processor 2376 stepping 02
    SMP alternatives: switching to SMP code
    Booting processor 2/8 APIC 0x1
    Initializing CPU#2
    Calibrating delay using timer specific routine.. 4589.75 BogoMIPS (lpj=2294876)
    CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
    CPU: L2 Cache: 512K (64 bytes/line)
    CPU 2/1 -> Node 0
    CPU: Physical Processor ID: 0
    CPU: Processor Core ID: 1
    Quad-Core AMD Opteron(tm) Processor 2376 stepping 02
    SMP alternatives: switching to SMP code
    Booting processor 3/8 APIC 0x5
    Initializing CPU#3
    Calibrating delay using timer specific routine.. 4588.55 BogoMIPS (lpj=2294276)
    CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
    CPU: L2 Cache: 512K (64 bytes/line)
    CPU 3/5 -> Node 1
    CPU: Physical Processor ID: 1
    CPU: Processor Core ID: 1
    Quad-Core AMD Opteron(tm) Processor 2376 stepping 02
    SMP alternatives: switching to SMP code
    Booting processor 4/8 APIC 0x2
    Initializing CPU#4
    Calibrating delay using timer specific routine.. 4589.53 BogoMIPS (lpj=2294766)
    CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
    CPU: L2 Cache: 512K (64 bytes/line)
    CPU 4/2 -> Node 0
    CPU: Physical Processor ID: 0
    CPU: Processor Core ID: 2
    Quad-Core AMD Opteron(tm) Processor 2376 stepping 02
    SMP alternatives: switching to SMP code
    Booting processor 5/8 APIC 0x6
    Initializing CPU#5
    Calibrating delay using timer specific routine.. 4588.56 BogoMIPS (lpj=2294282)
    CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
    CPU: L2 Cache: 512K (64 bytes/line)
    CPU 5/6 -> Node 1
    CPU: Physical Processor ID: 1
    CPU: Processor Core ID: 2
    Quad-Core AMD Opteron(tm) Processor 2376 stepping 02
    SMP alternatives: switching to SMP code
    Booting processor 6/8 APIC 0x3
    Initializing CPU#6
    Calibrating delay using timer specific routine.. 4588.54 BogoMIPS (lpj=2294272)
    CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
    CPU: L2 Cache: 512K (64 bytes/line)
    CPU 6/3 -> Node 0
    CPU: Physical Processor ID: 0
    CPU: Processor Core ID: 3
    Quad-Core AMD Opteron(tm) Processor 2376 stepping 02
    SMP alternatives: switching to SMP code
    Booting processor 7/8 APIC 0x7
    Initializing CPU#7
    Calibrating delay using timer specific routine.. 4589.48 BogoMIPS (lpj=2294740)
    CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
    CPU: L2 Cache: 512K (64 bytes/line)
    CPU 7/7 -> Node 1
    CPU: Physical Processor ID: 1
    CPU: Processor Core ID: 3
    Quad-Core AMD Opteron(tm) Processor 2376 stepping 02
    Brought up 8 CPUs
    CPU#0: NMI watchdog performance counter calibration - 579->599
    CPU#1: NMI watchdog performance counter calibration - 80->100
    CPU#2: NMI watchdog performance counter calibration - 77->97
    CPU#3: NMI watchdog performance counter calibration - 74->94
    CPU#4: NMI watchdog performance counter calibration - 71->91
    CPU#5: NMI watchdog performance counter calibration - 68->88
    CPU#6: NMI watchdog performance counter calibration - 64->84
    CPU#7: NMI watchdog performance counter calibration - 60->80
    NMI watchdog testing PASSED.
    time.c: Using 14.318180 MHz WALL HPET GTOD HPET/TSC timer.
    time.c: Detected 2294.251 MHz processor.
    sizeof(vma)=176 bytes
    sizeof(page)=56 bytes
    sizeof(inode)=560 bytes
    sizeof(dentry)=216 bytes
    sizeof(ext3inode)=760 bytes
    sizeof(buffer_head)=96 bytes
    sizeof(skbuff)=248 bytes
    migration_cost=81,4185
    checking if image is initramfs... it is
    Freeing initrd memory: 3379k freed
    NET: Registered protocol family 16
    ACPI: bus type pci registered
    Warning: pci_mmcfg_init marking 256MB space uncacheable.
    MCFG table requires 64MB uncacheable only. Try booting with acpi_mcfg_max_pci_bus_num=on
    PCI: Using MMCONFIG at f0000000
    ACPI: Interpreter enabled
    ACPI: Using IOAPIC for interrupt routing
    ACPI: No dock devices found.
    ACPI: PCI Root Bridge [PCI0] (0000:00)
    PCI: Enabling HT MSI Mapping on 0000:00:01.0
    ACPI: PCI Interrupt Routing Table [_SB_.PCI0._PRT]
    ACPI: PCI Interrupt Routing Table [_SB_.PCI0.PXB_._PRT]
    ACPI: PCI Interrupt Routing Table [_SB_.PCI0.PXB_.PPBX._PRT]
    ACPI: PCI Interrupt Routing Table [_SB_.PCI0.EXB0._PRT]
    ACPI: PCI Interrupt Routing Table [_SB_.PCI0.EXB1._PRT]
    ACPI: PCI Interrupt Routing Table [_SB_.PCI0.EXB2._PRT]
    ACPI: PCI Interrupt Routing Table [_SB_.PCI0.EXB3._PRT]
    ACPI: PCI Interrupt Routing Table [_SB_.PCI0.EXB4._PRT]
    ACPI: PCI Interrupt Link [LK10] (IRQs 3 4 5 6 7 10 11)
    ACPI: PCI Interrupt Link [LK11] (IRQs 3 4
    5 6 7 10 11)
    ACPI: PCI Interrupt Link [LK12] (IRQs 3 4 5 6 7 10 11) 0, disabled.
    ACPI: PCI Interrupt Link [LK13] (IRQs 3 4 5
    6 7 10 11)
    ACPI: PCI Interrupt Link [LK14] (IRQs 3 4 5 6 7 10 11)
    ACPI: PCI Interrupt Link [LK15] (IRQs 3 4 5 6 7
    10 11)
    ACPI: PCI Interrupt Link [LK16] (IRQs 3 4 5 6 7 10 11) 0, disabled.
    ACPI: PCI Interrupt Link [LK17] (IRQs 3 4 5 6 7 10 11)
    0, disabled.
    ACPI: PCI Interrupt Link [LK18] (IRQs 3 4 5 6 7 10 11) 0, disabled.
    ACPI: PCI Interrupt Link [LK19] (IRQs 3 4 5 6 7 10 11)
    0, disabled.
    ACPI: PCI Interrupt Link [LK1A] (IRQs 3 4 5 6 7 10 11) 0, disabled.
    ACPI: PCI Interrupt Link [LK1B] (IRQs 3 4 5 6 7 10 11)
    0, disabled.
    ACPI: PCI Interrupt Link [LK1C] (IRQs 3 4 5 6 7 10 11)
    ACPI: PCI Interrupt Link [LK1D] (IRQs 3 4 5 6 7 10 11)
    0, disabled.
    ACPI: PCI Interrupt Link [LK1E] (IRQs 3 4 5 6 7 10 11) 0, disabled.
    ACPI: PCI Interrupt Link [LK1F] (IRQs 3 4 5 6 7 10 11)
    0, disabled.
    ACPI: PCI Interrupt Link [LK20] (IRQs 3 4 5 6 7 10 11) 0, disabled.
    ACPI: PCI Interrupt Link [LK21] (IRQs 3 4 5 6 7 10 11)
    0, disabled.
    ACPI: PCI Interrupt Link [LK22] (IRQs 3 4 5 6 7 10 11) 0, disabled.
    ACPI: PCI Interrupt Link [LK23] (IRQs 3 4 5 6 7 10 11)
    0, disabled.
    ACPI: PCI Interrupt Link [LK24] (IRQs 3 4 5 6 7 10 11) 0, disabled.
    ACPI: PCI Interrupt Link [LK25] (IRQs 3 4 5 6 7 10 11)
    0, disabled.
    ACPI: PCI Interrupt Link [LK26] (IRQs 3 4 5 6 7 10 11) 0, disabled.
    ACPI: PCI Interrupt Link [LK27] (IRQs 3 4 5 6 7 10 11)
    0, disabled.
    ACPI: PCI Interrupt Link [LK28] (IRQs 3 4 5 6 7 10 11) 0, disabled.
    ACPI: PCI Interrupt Link [LK29] (IRQs 3 4 5 6 7 10 11)
    0, disabled.
    ACPI: PCI Interrupt Link [LK2A] (IRQs 3 4 5 6 7 10 11) 0, disabled.
    ACPI: PCI Interrupt Link [LK2B] (IRQs 3 4 5 6 7 10 11)
    0, disabled.
    ACPI: PCI Interrupt Link [LK2C] (IRQs 3 4 5 6 7 10 11) 0, disabled.
    ACPI: PCI Interrupt Link [LK2D] (IRQs 3 4 5 6 7 10 11)
    0, disabled.
    ACPI: PCI Interrupt Link [LK2E] (IRQs 3 4 5 6 7 10 11) 0, disabled.
    ACPI: PCI Interrupt Link [LK2F] (IRQs 3 4 5 6 7 10 11)
    0, disabled.
    ACPI: PCI Interrupt Link [LK01] (IRQs 3 4 5 6 7 10 11 12 14)
    ACPI: PCI Interrupt Link [LK07] (IRQs 3 4 5
    6 7 10 11 12 14)
    Linux Plug and Play Support v0.97 (c) Adam Belay
    pnp: PnP ACPI init
    pnp: PnP ACPI: found 11 devices
    usbcore: registered new driver usbfs
    usbcore: registered new driver hub
    PCI: Using ACPI for IRQ routing
    PCI: If a device doesn't work, try "pci=routeirq". If it helps, post a report
    NetLabel: Initializing
    NetLabel: domain hash size = 128
    NetLabel: protocols = UNLABELED CIPSOv4
    NetLabel: unlabeled traffic allowed by default
    hpet0: at MMIO 0xfed00000 (virtual 0xffffffffff5fe000), IRQs 2, 8, 0
    hpet0: 3 64-bit timers, 14318180 Hz
    ACPI: DMAR not present
    PCI-DMA: Disabling AGP.
    PCI-DMA: aperture base @ f4000000 size 65536 KB
    PCI-DMA: using GART IOMMU.
    PCI-DMA: Reserving 64MB of IOMMU area in the AGP aperture
    pnp: 00:07: ioport range 0x800-0x87f could not be reserved
    pnp: 00:07: ioport range 0x900-0x900 has been reserved
    pnp: 00:07: ioport range 0x880-0x89f has been reserved
    pnp: 00:07: ioport range 0xcd6-0xcd7 has been reserved
    pnp: 00:07: ioport range 0x40b-0x40b has been reserved
    pnp: 00:07: ioport range 0x4d6-0x4d6 has been reserved
    pnp: 00:08: ioport range 0xca8-0xca8 has been reserved
    pnp: 00:08: ioport range 0xcac-0xcac has been reserved
    pnp: 00:09: iomem range 0xf0000000-0xf1ffffff could not be reserved
    PCI: Bridge: 0000:03:0d.0
    IO window: disabled.
    MEM window: disabled.
    PREFETCH window: disabled.
    PCI: Bridge: 0000:00:01.0
    IO window: e000-efff
    MEM window: efc00000-efdfffff
    PREFETCH window 0x00000000e8000000-0x00000000e80fffff
    PCI: Bridge: 0000:00:07.0
    IO window: disabled.
    MEM window: disabled.
    PREFETCH window: disabled.
    PCI: Bridge: 0000:00:08.0
    IO window: disabled.
    MEM window: efe00000-efefffff
    PREFETCH window: disabled.
    PCI: Bridge: 0000:00:09.0
    IO window: disabled.
    MEM window: eff00000-efffffff
    PREFETCH window: disabled.
    PCI: Bridge: 0000:00:0a.0
    IO window: disabled.
    MEM window: disabled.
    PREFETCH window: disabled.
    PCI: Bridge: 0000:00:0b.0
    IO window: disabled.
    MEM window: disabled.
    PREFETCH window: disabled.
    GSI 16 sharing vector 0xA9 and IRQ 16
    ACPI: PCI Interrupt 0000:00:07.0[A] -> GSI 32 (level, low) -> IRQ 169
    PCI: Setting latency timer of device 0000:00:07.0 to 64
    GSI 17 sharing vector 0xB1 and IRQ 17
    ACPI: PCI Interrupt 0000:00:08.0[A] -> GSI 33 (level, low) -> IRQ 177
    PCI: Setting latency timer of device 0000:00:08.0 to 64
    GSI 18 sharing vector 0xB9 and IRQ 18
    ACPI: PCI Interrupt 0000:00:09.0[A] -> GSI 37 (level, low) -> IRQ 185
    PCI: Setting latency timer of device 0000:00:09.0 to 64
    GSI 19 sharing vector 0xC1 and IRQ 19
    ACPI: PCI Interrupt 0000:00:0a.0[A] -> GSI 35 (level, low) -> IRQ 193
    PCI: Setting latency timer of device 0000:00:0a.0 to 64
    GSI 20 sharing vector 0xC9 and IRQ 20
    ACPI: PCI Interrupt 0000:00:0b.0[A] -> GSI 36 (level, low) -> IRQ 201
    PCI: Setting latency timer of device 0000:00:0b.0 to 64
    NET: Registered protocol family 2
    IP route cache hash table entries: 262144 (order: 9, 2097152 bytes)
    TCP established hash table entries: 262144 (order: 10, 4194304 bytes)
    TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
    TCP: Hash tables configured (established 262144 bind 65536)
    TCP reno registered
    audit: initializing netlink socket (disabled)
    type=2000 audit(1308844914.974:1): initialized
    Total HugeTLB memory allocated, 0
    VFS: Disk quotas dquot_6.5.1
    Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
    SELinux: Registering netfilter hooks
    Initializing Cryptographic API
    alg: No test for crc32c (crc32c-generic)
    ksign: Installing public key data
    Loading keyring
    - Added public key 39EF749B8C2298D3
    - User ID: CentOS (Kernel Module GPG key)
    io scheduler noop registered
    io scheduler anticipatory registered
    io scheduler deadline registered
    io scheduler cfq registered (default)
    Boot video device is 0000:00:04.0
    PCI: Setting latency timer of device 0000:00:07.0 to 64
    PCI: Setting latency timer of device 0000:00:08.0 to 64
    PCI: Setting latency timer of device 0000:00:09.0 to 64
    PCI: Setting latency timer of device 0000:00:0a.0 to 64
    PCI: Setting latency timer of device 0000:00:0b.0 to 64
    pci_hotplug: PCI Hot Plug PCI Core version: 0.5
    Real Time Clock Driver v1.12ac
    hpet_resources: 0xfed00000 is busy
    Non-volatile memory driver v1.2
    Linux agpgart interface v0.101 (c) Dave Jones
    Serial: 8250/16550 driver $Revision: 1.90 $ 4 ports, IRQ sharing enabled
    serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
    serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
    00:05: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
    00:06: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
    brd: module loaded
    Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
    ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
    Probing IDE interface ide0...
    Probing IDE interface ide1...
    ide-floppy driver 0.99.newide
    usbcore: registered new driver hiddev
    usbcore: registered new driver usbhid
    drivers/usb/input/hid-core.c: v2.6:USB HID core driver
    PNP: No PS/2 controller found. Probing ports directly.
    serio: i8042 KBD port at 0x60,0x64 irq 1
    serio: i8042 AUX port at 0x60,0x64 irq 12
    mice: PS/2 mouse device common for all mice
    md: md driver 0.90.3 MAX_MD_DEVS=256, MD_SB_DISKS=27
    md: bitmap version 4.39
    TCP bic registered
    Initializing IPsec netlink socket
    NET: Registered protocol family 1
    NET: Registered protocol family 17
    ACPI: (supports S0 S4 S5)
    Initalizing network drop monitor service
    Freeing unused kernel memory: 224k freed
    Write protecting the kernel read-only data: 520k
    ACPI: PCI Interrupt Link [LK01] enabled at IRQ 11
    ACPI: PCI Interrupt 0000:00:03.2[A] -> Link [LK01] -> GSI 11 (level, low) -> IRQ 11
    ehci_hcd 0000:00:03.2: EHCI Host Controller
    ehci_hcd 0000:00:03.2: new USB bus registered, assigned bus number 1
    ehci_hcd 0000:00:03.2: irq 11, io mem 0xefbef000
    ehci_hcd 0000:00:03.2: USB 2.0 started, EHCI 1.00, driver 10 Dec 2004
    usb usb1: configuration #1 chosen from 1 choice
    hub 1-0:1.0: USB hub found
    hub 1-0:1.0: 4 ports detected
    ohci_hcd: 2005 April 22 USB 1.1 'Open' Host Controller (OHCI) Driver (PCI)
    ACPI: PCI Interrupt 0000:00:03.0[A] -> Link [LK01] -> GSI 11 (level, low) -> IRQ 11
    ohci_hcd 0000:00:03.0: OHCI Host Controller
    ohci_hcd 0000:00:03.0: new USB bus registered, assigned bus number 2
    ohci_hcd 0000:00:03.0: irq 11, io mem 0xefbed000
    usb usb2: configuration #1 chosen from 1 choice
    hub 2-0:1.0: USB hub found
    hub 2-0:1.0: 2 ports detected
    ACPI: PCI Interrupt 0000:00:03.1[A] -> Link [LK01] -> GSI 11 (level, low) -> IRQ 11
    ohci_hcd 0000:00:03.1: OHCI Host Controller
    ohci_hcd 0000:00:03.1: new USB bus registered, assigned bus number 3
    ohci_hcd 0000:00:03.1: irq 11, io mem 0xefbee000
    usb 1-3: new high speed USB device using ehci_hcd and address 2
    usb usb3: configuration #1 chosen from 1 choice
    hub 3-0:1.0: USB hub found
    hub 3-0:1.0: 2 ports detected
    usb 1-3: configuration #1 chosen from 1 choice
    hub 1-3:1.0: USB hub found
    hub 1-3:1.0: 2 ports detected
    USB Universal Host Controller Interface driver v3.0
    SCSI subsystem initialized
    libata version 3.00 loaded.
    sata_svw 0000:03:0e.0: version 2.3
    ACPI: PCI Interrupt Link [LK07] enabled at IRQ 10
    ACPI: PCI Interrupt 0000:03:0e.0[A] -> Link [LK07] -> GSI 10 (level, low) -> IRQ 10
    scsi0 : sata_svw
    scsi1 : sata_svw
    scsi2 : sata_svw
    scsi3 : sata_svw
    ata1: SATA max UDMA/133 mmio m8192@0xefdfe000 port 0xefdfe000 irq 10
    ata2: SATA max UDMA/133 mmio m8192@0xefdfe000 port 0xefdfe100 irq 10
    ata3: SATA max UDMA/133 mmio m8192@0xefdfe000 port 0xefdfe200 irq 10
    ata4: SATA max UDMA/133 mmio m8192@0xefdfe000 port 0xefdfe300 irq 10
    kobject_add failed for usbdev1.2_ep81 with -EEXIST, don't try to register things with the same name in the same directory.

    Call Trace:
    [<ffffffff801543f7>] kobject_add+0x166/0x191
    [<ffffffff801ca101>] device_add+0x85/0x372
    [<ffffffff801ffe3e>] usb_create_ep_files+0x137/0x19a
    [<ffffffff801ff8da>] usb_create_sysfs_intf_files+0x80/0x93
    [<ffffffff801fd1ac>] usb_set_configuration+0x3aa/0x3d9
    [<ffffffff801f8ec4>] usb_new_device+0x253/0x2c4
    [<ffffffff801f9ffc>] hub_thread+0x742/0xb01
    [<ffffffff800a28ec>] autoremove_wake_function+0x0/0x2e
    [<ffffffff801f98ba>] hub_thread+0x0/0xb01
    [<ffffffff800a26d4>] keventd_create_kthread+0x0/0xc4
    [<ffffffff80032b26>] kthread+0xfe/0x132
    [<ffffffff8005dfb1>] child_rip+0xa/0x11
    [<ffffffff800a26d4>] keventd_create_kthread+0x0/0xc4
    [<ffffffff80032a28>] kthread+0x0/0x132
    [<ffffffff8005dfa7>] child_rip+0x0/0x11

    ata1: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
    ata1.00: ATA-8: WDC WD2502ABYS-18B7A0, 02.03B04, max UDMA/133
    ata1.00: 488281250 sectors, multi 8: LBA48 NCQ (depth 0/32)
    usb 1-3.2: new low speed USB device using ehci_hcd and address 3
    ata1.00: configured for UDMA/133
    usb 1-3.2: configuration #1 chosen from 1 choice
    input: Logitech Logitech USB Keyboard as /class/input/input0
    input: USB HID v1.10 Keyboard [Logitech Logitech USB Keyboard] on usb-0000:00:03.2-3.2
    ata2: SATA link down (SStatus 4 SControl 300)
    ata3: SATA link down (SStatus 4 SControl 300)
    ata4: SATA link down (SStatus 4 SControl 300)
    Vendor: ATA Model: WDC WD2502ABYS-1 Rev: 02.0
    Type: Direct-Access ANSI SCSI revision: 05
    SCSI device sda: 488281250 512-byte hdwr sectors (250000 MB)
    sda: Write Protect is off
    sda: Mode Sense: 00 3a 00 00
    SCSI device sda: drive cache: write back
    SCSI device sda: 488281250 512-byte hdwr sectors (250000 MB)
    sda: Write Protect is off
    sda: Mode Sense: 00 3a 00 00
    SCSI device sda: drive cache: write back
    sda: sda1 sda2
    sd 0:0:0:0: Attached scsi disk sda
    shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
    device-mapper: uevent: version 1.0.3
    device-mapper: ioctl: 4.11.5-ioctl (2007-12-12) initialised: dm-devel@redhat.com
    device-mapper: dm-raid45: initialized v0.2594l
    kjournald starting. Commit interval 5 seconds
    EXT3-fs: mounted filesystem with ordered data mode.
    security: 3 users, 6 roles, 1991 types, 265 bools, 1 sens, 1024 cats
    security: 61 classes, 78571 rules
    SELinux: Completing initialization.
    SELinux: Setting up existing superblocks.
    SELinux: initialized (dev dm-0, type ext3), uses xattr
    SELinux: initialized (dev usbfs, type usbfs), uses genfs_contexts
    SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs
    SELinux: initialized (dev debugfs, type debugfs), uses genfs_contexts
    SELinux: initialized (dev selinuxfs, type selinuxfs), uses genfs_contexts
    SELinux: initialized (dev mqueue, type mqueue), uses transition SIDs
    SELinux: initialized (dev hugetlbfs, type hugetlbfs), uses genfs_contexts
    SELinux: initialized (dev devpts, type devpts), uses transition SIDs
    SELinux: initialized (dev eventpollfs, type eventpollfs), uses task SIDs
    SELinux: initialized (dev inotifyfs, type inotifyfs), uses genfs_contexts
    SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs
    SELinux: initialized (dev futexfs, type futexfs), uses genfs_contexts
    SELinux: initialized (dev anon_inodefs, type anon_inodefs), uses genfs_contexts
    SELinux: initialized (dev pipefs, type pipefs), uses task SIDs
    SELinux: initialized (dev sockfs, type sockfs), uses task SIDs
    SELinux: initialized (dev cpuset, type cpuset), uses genfs_contexts
    SELinux: initialized (dev proc, type proc), uses genfs_contexts
    SELinux: initialized (dev bdev, type bdev), uses genfs_contexts
    SELinux: initialized (dev rootfs, type rootfs), uses genfs_contexts
    SELinux: initialized (dev sysfs, type sysfs), uses genfs_contexts
    type=1403 audit(1308844939.372:2): policy loaded auid=4294967295 ses=4294967295
    type=1400 audit(1308844940.133:3): avc: denied { read } for pid=759 comm="restorecon" name="libc.so.6" dev=dm-0 ino=10649904 scontext=system_u:system_r:restorecon_t:s0 tcontext=system_u:object_r:file_t:s0 tclass=lnk_file
    type=1400 audit(1308844940.134:4): avc: denied { execute } for pid=759 comm="restorecon" path="/lib64/libc-2.5.so" dev=dm-0 ino=10649610 scontext=system_u:system_r:restorecon_t:s0 tcontext=system_u:object_r:file_t:s0 tclass=file
    type=1400 audit(1308844940.389:5): avc: denied { read } for pid=767 comm="hwclock" name="ld.so.cache" dev=dm-0 ino=28443742 scontext=system_u:system_r:hwclock_t:s0 tcontext=system_u:object_r:file_t:s0 tclass=file
    type=1400 audit(1308844940.390:6): avc: denied { getattr } for pid=767 comm="hwclock" path="/etc/ld.so.cache" dev=dm-0 ino=28443742 scontext=system_u:system_r:hwclock_t:s0 tcontext=system_u:object_r:file_t:s0 tclass=file
    type=1400 audit(1308844940.402:7): avc: denied { read } for pid=767 comm="hwclock" name="libc.so.6" dev=dm-0 ino=10649904 scontext=system_u:system_r:hwclock_t:s0 tcontext=system_u:object_r:file_t:s0 tclass=lnk_file
    type=1400 audit(1308844940.403:8): avc: denied { execute } for pid=767 comm="hwclock" path="/lib64/libc-2.5.so" dev=dm-0 ino=10649610 scontext=system_u:system_r:hwclock_t:s0 tcontext=system_u:object_r:file_t:s0 tclass=file
    type=1400 audit(1308880942.866:9): avc: denied { read } for pid=1249 comm="pam_console_app" name="ld.so.cache" dev=dm-0 ino=28443742 scontext=system_u:system_r:pam_console_t:s0-s0:c0.c1023 tcontext=system_u:object_r:file_t:s0 tclass=file
    type=1400 audit(1308880942.866:10): avc: denied { read } for pid=1266 comm="pam_console_app" name="ld.so.cache" dev=dm-0 ino=28443742 scontext=system_u:system_r:pam_console_t:s0-s0:c0.c1023 tcontext=system_u:object_r:file_t:s0 tclass=file
    type=1400 audit(1308880942.866:11): avc: denied { read } for pid=1252 comm="pam_console_app" name="ld.so.cache" dev=dm-0 ino=28443742 scontext=system_u:system_r:pam_console_t:s0-s0:c0.c1023 tcontext=system_u:object_r:file_t:s0 tclass=file
    input: PC Speaker as /class/input/input1
    EDAC MC: Ver: 2.0.1 May 31 2011
    tg3.c:v3.108 (February 17, 2010)
    ACPI: PCI Interrupt 0000:01:00.0[A] -> GSI 33 (level, low) -> IRQ 177
    PCI: Setting latency timer of device 0000:01:00.0 to 64
    eth0: Tigon3 [partno(BCM95721) rev 4201 PHY(5750)] (PCI Express) 10/100/1000Base-T Ethernet 00:22:19:22:18:9c
    eth0: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[1] WireSpeed[1] TSOcap[1]
    eth0: dma_rwctrl[76180000] dma_mask[64-bit]
    ACPI: PCI Interrupt 0000:02:00.0[A] -> GSI 37 (level, low) -> IRQ 185
    PCI: Setting latency timer of device 0000:02:00.0 to 64
    eth1: Tigon3 [partno(BCM95721) rev 4201 PHY(5750)] (PCI Express) 10/100/1000Base-T Ethernet 00:22:19:22:18:9d
    eth1: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[1] WireSpeed[1] TSOcap[1]
    eth1: dma_rwctrl[76180000] dma_mask[64-bit]
    k10temp 0000:00:18.3: unreliable CPU thermal sensor; monitoring disabled
    k10temp 0000:00:19.3: unreliable CPU thermal sensor; monitoring disabled
    piix4_smbus 0000:00:02.0: Found 0000:00:02.0 device
    EDAC amd64_edac: Ver: 3.2.0 May 31 2011
    EDAC amd64: ECC is enabled by BIOS.
    EDAC amd64: ECC is enabled by BIOS.
    EDAC MC: F10h CPU detected
    EDAC MC0: Giving out device to amd64_edac Family 10h: DEV 0000:00:18.2
    EDAC MC: F10h CPU detected
    EDAC MC1: Giving out device to amd64_edac Family 10h: DEV 0000:00:19.2
    sd 0:0:0:0: Attached scsi generic sg0 type 0
    floppy0: no floppy controllers found
    work still pending
    lp: driver loaded but no devices found
    ACPI: Power Button (FF) [PWRF]
    ACPI: Mapper loaded
    dell-wmi: No known WMI GUID found
    md: Autodetecting RAID arrays.
    md: autorun ...
    md: ... autorun DONE.
    device-mapper: multipath: version 1.0.6 loaded
    loop: loaded (max 8 devices)
    printk: 21 messages suppressed.
    type=1400 audit(1308880949.770:19): avc: denied { read } for pid=2057 comm="pam_console_app" name="ld.so.cache" dev=dm-0 ino=28443742 scontext=system_u:system_r:pam_console_t:s0-s0:c0.c1023 tcontext=system_u:object_r:file_t:s0 tclass=file
    EXT3 FS on dm-0, internal journal
    kjournald starting. Commit interval 5 seconds
    EXT3 FS on sda1, internal journal
    EXT3-fs: mounted filesystem with ordered data mode.
    SELinux: initialized (dev sda1, type ext3), uses xattr
    SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs
    SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs
    printk: 9 messages suppressed.
    type=1400 audit(1308880950.341:23): avc: denied { execute } for pid=2077 comm="restorecon" path="/lib64/libc-2.5.so" dev=dm-0 ino=10649610 scontext=system_u:system_r:restorecon_t:s0 tcontext=system_u:object_r:file_t:s0 tclass=file
    Adding 10289144k swap on /dev/VolGroup00/LogVol01. Priority:-1 extents:1 across:10289144k
    SELinux: initialized (dev binfmt_misc, type binfmt_misc), uses genfs_contexts
    Loading iSCSI transport class v2.0-871.
    802.1Q VLAN Support v1.8 Ben Greear greearb@candelatech.com
    All bugs added by David S. Miller davem@redhat.com
    cxgb3i: tag itt 0x1fff, 13 bits, age 0xf, 4 bits.
    iscsi: registered transport (cxgb3i)
    NET: Registered protocol family 10
    lo: Disabled Privacy Extensions
    IPv6 over IPv4 tunneling driver
    Broadcom NetXtreme II CNIC Driver cnic v2.1.2 (May 26, 2010)
    Broadcom NetXtreme II iSCSI Driver bnx2i v2.1.3 (Aug 10, 2010)
    iscsi: registered transport (bnx2i)
    iscsi: registered transport (tcp)
    iscsi: registered transport (iser)
    iscsi: registered transport (be2iscsi)
    ADDRCONF(NETDEV_UP): eth0: link is not ready
    tg3: eth0: Link is up at 100 Mbps, full duplex.
    tg3: eth0: Flow control is off for TX and off for RX.
    ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
    printk: 66 messages suppressed.
    type=1400 audit(1308880958.905:46): avc: denied { read } for pid=2797 comm="auditd" name="libnsl.so.1" dev=dm-0 ino=10649908 scontext=system_u:system_r:auditd_t:s0 tcontext=system_u:object_r:file_t:s0 tclass=lnk_file
    powernow-k8: Found 2 Quad-Core AMD Opteron(tm) Processor 2376 processors (8 cpu cores) (version 2.20.00)
    powernow-k8: 0 : fid 0x0 gid 0x0 (2300 MHz)
    powernow-k8: 1 : fid 0x0 gid 0x0 (1600 MHz)
    powernow-k8: 2 : fid 0x0 gid 0x0 (1100 MHz)
    powernow-k8: 3 : fid 0x0 gid 0x0 (800 MHz)
    powernow-k8: 0 : fid 0x0 gid 0x0 (2300 MHz)
    powernow-k8: 1 : fid 0x0 gid 0x0 (1600 MHz)
    powernow-k8: 2 : fid 0x0 gid 0x0 (1100 MHz)
    powernow-k8: 3 : fid 0x0 gid 0x0 (800 MHz)
    SELinux: initialized (dev rpc_pipefs, type rpc_pipefs), uses genfs_contexts
    eth0: no IPv6 routers present
    FS-Cache: Loaded
    SELinux: initialized (dev 0:17, type nfs), uses genfs_contexts
    ADDRCONF(NETDEV_UP): eth1: link is not ready
    tg3: eth1: Link is up at 100 Mbps, full duplex.
    tg3: eth1: Flow control is off for TX and off for RX.
    ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
    eth1: no IPv6 routers present
    tg3: eth1: Link is down.
    tg3: eth1: Link is up at 1000 Mbps, full duplex.
    tg3: eth1: Flow control is on for TX and on for RX.
    device eth0 entered promiscuous mode
    device eth0 left promiscuous mode
    device eth1 entered promiscuous mode
    device eth1 left promiscuous mode
    SELinux: initialized (dev 0:17, type nfs), uses genfs_contexts

     
  • Comment has been marked as spam. 
    Undo

    You can see all pending comments posted by this user  here

    Anonymous - 2011-06-30

    I'd be more than happy to use the debug driver. Ideally I wouldn't like to mess about with hand-rolling the kernel for the boxes given they're our most outside accessible servers, but if it's really necessary we can do.

     
  • Tushar Dave

    Tushar Dave - 2011-06-30

    By looking at the output of ethtool and dmesg , it looks like they are from other vendor's Network card/interface (e.g tg3). Its not Intel e1000 driver.
    Could you please make sure and resend the output as requested before?

     
  • Comment has been marked as spam. 
    Undo

    You can see all pending comments posted by this user  here

    Anonymous - 2011-06-30

    From lshw's output:

           *-pci:0
                description: PCI bridge
                product: 6700PXH PCI Express-to-PCI Bridge A
                vendor: Intel Corporation
                physical id: 0
                bus info: pci@0000:01:00.0
                version: 09
                width: 32 bits
                clock: 33MHz
                capabilities: pci pciexpress msi pm pcix normal_decode bus_master cap_list
              *-network
                   description: Ethernet interface
                   product: 82541GI Gigabit Ethernet Controller
                   vendor: Intel Corporation
                   physical id: 4
                   bus info: pci@0000:02:04.0
                   logical name: eth0
                   version: 05
                   serial: 00:15:c5:5d:2d:8e
                   size: 100MB/s
                   capacity: 1GB/s
                   width: 32 bits
                   clock: 66MHz
                   capabilities: pm pcix bus_master cap_list ethernet physical tp 10bt 10bt-fd 100bt 100bt-fd 1000bt-fd autonegotiation
                   configuration: autonegotiation=on broadcast=yes driver=e1000 driverversion=8.0.30-NAPI duplex=full firmware=N/A ip=**.**.**.** latency=32 link=yes mingnt=255 module=e1000 multicast=yes port=twisted pair speed=100MB/
    
     
  • Comment has been marked as spam. 
    Undo

    You can see all pending comments posted by this user  here

    Anonymous - 2011-07-01

    Hmm.. just noticed what you mean about ethtool output. How bizarre. I think I must have logged into the wrong box by mistake or something.

    Let's try all that again.

    Linux karihi.ehawaii.gov 2.6.18-238.12.1.el5 #1 SMP Tue May 31 13:22:04 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux

           CPU0       CPU1
    

    0: 1107783292 0 IO-APIC-edge timer
    1: 3 0 IO-APIC-edge i8042
    4: 383 11375 IO-APIC-edge serial
    8: 1 0 IO-APIC-edge rtc
    9: 0 0 IO-APIC-level acpi
    12: 4 0 IO-APIC-edge i8042
    14: 592 0 IO-APIC-edge ide0
    169: 0 0 IO-APIC-level uhci_hcd:usb2
    177: 9847 6029117 IO-APIC-level ata_piix
    185: 1 0 IO-APIC-level ehci_hcd:usb1
    193: 0 0 IO-APIC-level uhci_hcd:usb3
    201: 4004 66679673 IO-APIC-level eth0
    209: 1000023 0 IO-APIC-level eth1
    NMI: 49855 49649
    LOC: 1107808476 1107742039
    ERR: 0
    MIS: 0

    driver: e1000
    version: 8.0.30-NAPI
    firmware-version: N/A
    bus-info: 0000:02:04.0

    NIC statistics:
    rx_packets: 1340259
    tx_packets: 1594805
    rx_bytes: 321421888
    tx_bytes: 272672976
    rx_broadcast: 573
    tx_broadcast: 222
    rx_multicast: 1533
    tx_multicast: 204728
    rx_errors: 0
    tx_errors: 0
    tx_dropped: 0
    multicast: 1533
    collisions: 0
    rx_length_errors: 0
    rx_over_errors: 0
    rx_crc_errors: 0
    rx_frame_errors: 0
    rx_no_buffer_count: 0
    rx_missed_errors: 0
    tx_aborted_errors: 0
    tx_carrier_errors: 0
    tx_fifo_errors: 0
    tx_heartbeat_errors: 0
    tx_window_errors: 0
    tx_abort_late_coll: 0
    tx_deferred_ok: 0
    tx_single_coll_ok: 0
    tx_multi_coll_ok: 0
    tx_timeout_count: 0
    tx_restart_queue: 0
    rx_long_length_errors: 0
    rx_short_length_errors: 0
    rx_align_errors: 0
    tx_tcp_seg_good: 9431
    tx_tcp_seg_failed: 0
    rx_flow_control_xon: 0
    rx_flow_control_xoff: 0
    tx_flow_control_xon: 0
    tx_flow_control_xoff: 0
    rx_long_byte_count: 321421888
    rx_csum_offload_good: 1330754
    rx_csum_offload_errors: 0
    alloc_rx_buff_failed: 0
    tx_smbus: 0
    rx_smbus: 0
    dropped_smbus: 0

    Linux version 2.6.18-238.12.1.el5 (mockbuild@builder10.centos.org) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-50)) #1 SMP Tue May 31 13:22:04 EDT 2011
    Command line: ro root=/dev/vg0/root console=ttyS0,9600
    BIOS-provided physical RAM map:
    BIOS-e820: 0000000000010000 - 00000000000a0000 (usable)
    BIOS-e820: 0000000000100000 - 00000000bffc0000 (usable)
    BIOS-e820: 00000000bffc0000 - 00000000bffcfc00 (ACPI data)
    BIOS-e820: 00000000bffcfc00 - 00000000bffff000 (reserved)
    BIOS-e820: 00000000e0000000 - 00000000f0000000 (reserved)
    BIOS-e820: 00000000fec00000 - 00000000fec90000 (reserved)
    BIOS-e820: 00000000fed00000 - 00000000fed00400 (reserved)
    BIOS-e820: 00000000fee00000 - 00000000fee10000 (reserved)
    BIOS-e820: 00000000ffb00000 - 0000000100000000 (reserved)
    DMI 2.3 present.
    ACPI: RSDP (v000 DELL ) @ 0x00000000000fd650
    ACPI: RSDT (v001 DELL PESC1425 0x00000001 MSFT 0x0100000a) @ 0x00000000000fd664
    ACPI: FADT (v001 DELL PESC1425 0x00000001 MSFT 0x0100000a) @ 0x00000000000fd6b0
    ACPI: MADT (v001 DELL PESC1425 0x00000001 MSFT 0x0100000a) @ 0x00000000000fd724
    ACPI: SPCR (v001 DELL PESC1425 0x00000001 MSFT 0x0100000a) @ 0x00000000000fd7c0
    ACPI: HPET (v001 DELL PESC1425 0x00000001 MSFT 0x0100000a) @ 0x00000000000fd810
    ACPI: MCFG (v001 DELL PESC1425 0x00000001 MSFT 0x0100000a) @ 0x00000000000fd848
    ACPI: DSDT (v001 DELL PESC1425 0x00000001 MSFT 0x0100000e) @ 0x0000000000000000
    No NUMA configuration found
    Faking a node at 0000000000000000-00000000bffc0000
    Bootmem setup node 0 0000000000000000-00000000bffc0000
    bootmap start 1b000 pages 24
    Memory for crash kernel (0x0 to 0x0) notwithin permissible range
    disabling kdump
    Setting up node 0 0-bffc0
    On node 0 totalpages: 774016
    DMA zone: 2439 pages, LIFO batch:0
    DMA32 zone: 771577 pages, LIFO batch:31
    ACPI: PM-Timer IO Port: 0x808
    ACPI: Local APIC address 0xfee00000
    ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
    Processor #0 15:4 APIC version 20
    ACPI: LAPIC (acpi_id[0x02] lapic_id[0x01] enabled)
    Processor #1 15:4 APIC version 20
    ACPI: LAPIC (acpi_id[0x03] lapic_id[0x06] disabled)
    ACPI: LAPIC (acpi_id[0x04] lapic_id[0x07] disabled)
    ACPI: LAPIC_NMI (acpi_id[0x01] high edge lint[0x1])
    ACPI: LAPIC_NMI (acpi_id[0x02] high edge lint[0x1])
    ACPI: LAPIC_NMI (acpi_id[0x03] high edge lint[0x1])
    ACPI: LAPIC_NMI (acpi_id[0x04] high edge lint[0x1])
    ACPI: IOAPIC (id[0x02] address[0xfec00000] gsi_base[0])
    IOAPIC[0]: apic_id 2, version 32, address 0xfec00000, GSI 0-23
    ACPI: IOAPIC (id[0x03] address[0xfec80000] gsi_base[32])
    IOAPIC[1]: apic_id 3, version 32, address 0xfec80000, GSI 32-55
    ACPI: IOAPIC (id[0x04] address[0xfec80800] gsi_base[64])
    IOAPIC[2]: apic_id 4, version 32, address 0xfec80800, GSI 64-87
    ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
    ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
    ACPI: IRQ0 used by override.
    ACPI: IRQ2 used by override.
    ACPI: IRQ9 used by override.
    Setting APIC routing to physical flat
    ACPI: HPET id: 0xffffffff base: 0xfed00000
    Using ACPI (MADT) for SMP configuration information
    Nosave address range: 00000000000a0000 - 0000000000100000
    Allocating PCI resources starting at c0000000 (gap: bffff000:20001000)
    SMP: Allowing 4 CPUs, 2 hotplug CPUs
    Built 1 zonelists. Total pages: 774016
    Kernel command line: ro root=/dev/vg0/root console=ttyS0,9600
    Initializing CPU#0
    PID hash table entries: 4096 (order: 12, 32768 bytes)
    Console: colour VGA+ 80x25
    Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes)
    Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes)
    Checking aperture...
    ACPI: DMAR not present
    Memory: 3086088k/3145472k available (2592k kernel code, 58936k reserved, 1653k data, 224k init)
    Calibrating delay loop (skipped), value calculated using timer frequency.. 5600.24 BogoMIPS (lpj=2800124)
    Security Framework v1.0.0 initialized
    SELinux: Initializing.
    SELinux: Starting in permissive mode
    selinux_register_security: Registering secondary module capability
    Capability LSM initialized as secondary
    Mount-cache hash table entries: 256
    CPU: Trace cache: 12K uops, L1 D cache: 16K
    CPU: L2 cache: 2048K
    using mwait in idle threads.
    CPU: Physical Processor ID: 0
    CPU: Processor Core ID: 0
    CPU0: Thermal monitoring enabled (TM1)
    SMP alternatives: switching to UP code
    ACPI: Core revision 20060707
    Using local APIC timer interrupts.
    Detected 12.500 MHz APIC timer.
    SMP alternatives: switching to SMP code
    Booting processor 1/2 APIC 0x1
    Initializing CPU#1
    Calibrating delay using timer specific routine.. 5600.32 BogoMIPS (lpj=2800162)
    CPU: Trace cache: 12K uops, L1 D cache: 16K
    CPU: L2 cache: 2048K
    CPU: Physical Processor ID: 0
    CPU: Processor Core ID: 0
    CPU1: Thermal monitoring enabled (TM1)
    Intel(R) Xeon(TM) CPU 2.80GHz stepping 0a
    Brought up 2 CPUs
    CPU#0: NMI watchdog performance counter calibration - 626->646
    CPU#1: NMI watchdog performance counter calibration - 420->440
    NMI watchdog testing PASSED.
    time.c: Using 14.318180 MHz WALL HPET GTOD HPET/TSC timer.
    time.c: Detected 2800.124 MHz processor.
    sizeof(vma)=176 bytes
    sizeof(page)=56 bytes
    sizeof(inode)=560 bytes
    sizeof(dentry)=216 bytes
    sizeof(ext3inode)=760 bytes
    sizeof(buffer_head)=96 bytes
    sizeof(skbuff)=248 bytes
    migration_cost=3
    checking if image is initramfs... it is
    Freeing initrd memory: 3366k freed
    NET: Registered protocol family 16
    ACPI: bus type pci registered
    PCI: Using MMCONFIG at e0000000
    ACPI: Interpreter enabled
    ACPI: Using IOAPIC for interrupt routing
    ACPI: No dock devices found.
    ACPI: PCI Root Bridge [PCI0] (0000:00)
    PCI quirk: region 0800-087f claimed by ICH4 ACPI/GPIO/TCO
    PCI quirk: region 0880-08bf claimed by ICH4 GPIO
    PCI: Ignoring BAR0-3 of IDE controller 0000:00:1f.1
    PCI: PXH quirk detected, disabling MSI for SHPC device
    PCI: PXH quirk detected, disabling MSI for SHPC device
    PCI: Transparent bridge - 0000:00:1e.0
    ACPI: PCI Interrupt Routing Table [_SB_.PCI0._PRT]
    ACPI: PCI Interrupt Routing Table [_SB_.PCI0.PALO._PRT]
    ACPI: PCI Interrupt Routing Table [_SB_.PCI0.PALO.PXHB._PRT]
    ACPI: PCI Interrupt Routing Table [_SB_.PCI0.PALO.PXHA._PRT]
    ACPI: PCI Interrupt Routing Table [_SB_.PCI0.PICH._PRT]
    ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 6 7 10 11 12)
    ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 5 6 7 10
    11 12)
    ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 5 6 7 10 11 12)
    ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 5 6 7
    10 11 12)
    ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 5 6 7 10 11 12)
    ACPI: PCI Interrupt Link [LNKF] (IRQs 3 4 5 6 7 10 11 12)
    0, disabled.
    ACPI: PCI Interrupt Link [LNKG] (IRQs 3 4 5 6 7 10 11 12) 0, disabled.
    ACPI: PCI Interrupt Link [LNKH] (IRQs 3 4 5
    6 7 10 11 12)
    Linux Plug and Play Support v0.97 (c) Adam Belay
    pnp: PnP ACPI init
    pnp: PnP ACPI: found 10 devices
    usbcore: registered new driver usbfs
    usbcore: registered new driver hub
    PCI: Using ACPI for IRQ routing
    PCI: If a device doesn't work, try "pci=routeirq". If it helps, post a report
    NetLabel: Initializing
    NetLabel: domain hash size = 128
    NetLabel: protocols = UNLABELED CIPSOv4
    NetLabel: unlabeled traffic allowed by default
    hpet0: at MMIO 0xfed00000 (virtual 0xffffffffff5fe000), IRQs 2, 8, 0
    hpet0: 3 64-bit timers, 14318180 Hz
    ACPI: DMAR not present
    PCI-GART: No AMD northbridge found.
    pnp: 00:06: ioport range 0x800-0x87f could not be reserved
    pnp: 00:06: ioport range 0x880-0x8bf has been reserved
    pnp: 00:06: ioport range 0x8c0-0x8df has been reserved
    pnp: 00:06: ioport range 0x8e0-0x8e3 has been reserved
    pnp: 00:06: ioport range 0xc00-0xc0f has been reserved
    pnp: 00:06: ioport range 0xc10-0xc1f has been reserved
    pnp: 00:06: ioport range 0xca0-0xca7 has been reserved
    pnp: 00:06: ioport range 0xca9-0xcab has been reserved
    pnp: 00:08: iomem range 0xe0000000-0xefffffff could not be reserved
    PCI: Bridge: 0000:01:00.0
    IO window: e000-efff
    MEM window: fe900000-feafffff
    PREFETCH window: disabled.
    PCI: Bridge: 0000:01:00.2
    IO window: disabled.
    MEM window: disabled.
    PREFETCH window: disabled.
    PCI: Bridge: 0000:00:02.0
    IO window: e000-efff
    MEM window: fe700000-feafffff
    PREFETCH window: disabled.
    PCI: Bridge: 0000:00:1e.0
    IO window: d000-dfff
    MEM window: fe500000-fe6fffff
    PREFETCH window 0x00000000f0000000-0x00000000f7ffffff
    GSI 16 sharing vector 0xA9 and IRQ 16
    ACPI: PCI Interrupt 0000:00:02.0[A] -> GSI 16 (level, low) -> IRQ 169
    PCI: Setting latency timer of device 0000:00:02.0 to 64
    PCI: Setting latency timer of device 0000:01:00.0 to 64
    PCI: Setting latency timer of device 0000:01:00.2 to 64
    PCI: Setting latency timer of device 0000:00:1e.0 to 64
    NET: Registered protocol family 2
    IP route cache hash table entries: 131072 (order: 8, 1048576 bytes)
    TCP established hash table entries: 262144 (order: 10, 4194304 bytes)
    TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
    TCP: Hash tables configured (established 262144 bind 65536)
    TCP reno registered
    audit: initializing netlink socket (disabled)
    type=2000 audit(1308373306.561:1): initialized
    Total HugeTLB memory allocated, 0
    VFS: Disk quotas dquot_6.5.1
    Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
    SELinux: Registering netfilter hooks
    Initializing Cryptographic API
    alg: No test for crc32c (crc32c-generic)
    ksign: Installing public key data
    Loading keyring
    - Added public key 39EF749B8C2298D3
    - User ID: CentOS (Kernel Module GPG key)
    io scheduler noop registered
    io scheduler anticipatory registered
    io scheduler deadline registered
    io scheduler cfq registered (default)
    Intel E7520/7320/7525 detected.<7>Boot video device is 0000:04:0d.0
    PCI: Setting latency timer of device 0000:00:02.0 to 64
    pci_hotplug: PCI Hot Plug PCI Core version: 0.5
    Real Time Clock Driver v1.12ac
    hpet_resources: 0xfed00000 is busy
    Non-volatile memory driver v1.2
    Linux agpgart interface v0.101 (c) Dave Jones
    Serial: 8250/16550 driver $Revision: 1.90 $ 4 ports, IRQ sharing enabled
    serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a NS16550A
    00:05: ttyS0 at I/O 0x3f8 (irq = 4) is a NS16550A
    brd: module loaded
    Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
    ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
    ICH5: IDE controller at PCI slot 0000:00:1f.1
    PCI: Enabling device 0000:00:1f.1 (0005 -> 0007)
    GSI 17 sharing vector 0xB1 and IRQ 17
    ACPI: PCI Interrupt 0000:00:1f.1[A] -> GSI 18 (level, low) -> IRQ 177
    ICH5: chipset revision 2
    ICH5: not 100% native mode: will probe irqs later
    ide0: BM-DMA at 0xfc00-0xfc07, BIOS settings: hda:DMA, hdb:pio
    Probing IDE interface ide0...
    hda: HL-DT-STCD-RW/DVD-ROM GCC-4244N, ATAPI CD/DVD-ROM drive
    ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
    Probing IDE interface ide1...
    ide-floppy driver 0.99.newide
    usbcore: registered new driver hiddev
    usbcore: registered new driver usbhid
    drivers/usb/input/hid-core.c: v2.6:USB HID core driver
    PNP: No PS/2 controller found. Probing ports directly.
    serio: i8042 KBD port at 0x60,0x64 irq 1
    serio: i8042 AUX port at 0x60,0x64 irq 12
    mice: PS/2 mouse device common for all mice
    md: md driver 0.90.3 MAX_MD_DEVS=256, MD_SB_DISKS=27
    md: bitmap version 4.39
    TCP bic registered
    Initializing IPsec netlink socket
    NET: Registered protocol family 1
    NET: Registered protocol family 17
    ACPI: (supports S0 S4 S5)
    Initalizing network drop monitor service
    Freeing unused kernel memory: 224k freed
    Write protecting the kernel read-only data: 520k
    GSI 18 sharing vector 0xB9 and IRQ 18
    ACPI: PCI Interrupt 0000:00:1d.7[D] -> GSI 23 (level, low) -> IRQ 185
    PCI: Setting latency timer of device 0000:00:1d.7 to 64
    ehci_hcd 0000:00:1d.7: EHCI Host Controller
    ehci_hcd 0000:00:1d.7: new USB bus registered, assigned bus number 1
    ehci_hcd 0000:00:1d.7: debug port 1
    PCI: cache line size of 128 is not supported by device 0000:00:1d.7
    ehci_hcd 0000:00:1d.7: irq 185, io mem 0xfeb00000
    ehci_hcd 0000:00:1d.7: USB 2.0 started, EHCI 1.00, driver 10 Dec 2004
    usb usb1: configuration #1 chosen from 1 choice
    hub 1-0:1.0: USB hub found
    hub 1-0:1.0: 4 ports detected
    ohci_hcd: 2005 April 22 USB 1.1 'Open' Host Controller (OHCI) Driver (PCI)
    USB Universal Host Controller Interface driver v3.0
    ACPI: PCI Interrupt 0000:00:1d.0[A] -> GSI 16 (level, low) -> IRQ 169
    PCI: Setting latency timer of device 0000:00:1d.0 to 64
    uhci_hcd 0000:00:1d.0: UHCI Host Controller
    uhci_hcd 0000:00:1d.0: new USB bus registered, assigned bus number 2
    uhci_hcd 0000:00:1d.0: irq 169, io base 0x0000cce0
    usb usb2: configuration #1 chosen from 1 choice
    hub 2-0:1.0: USB hub found
    hub 2-0:1.0: 2 ports detected
    GSI 19 sharing vector 0xC1 and IRQ 19
    ACPI: PCI Interrupt 0000:00:1d.1[B] -> GSI 19 (level, low) -> IRQ 193
    PCI: Setting latency timer of device 0000:00:1d.1 to 64
    uhci_hcd 0000:00:1d.1: UHCI Host Controller
    uhci_hcd 0000:00:1d.1: new USB bus registered, assigned bus number 3
    uhci_hcd 0000:00:1d.1: irq 193, io base 0x0000ccc0
    usb usb3: configuration #1 chosen from 1 choice
    hub 3-0:1.0: USB hub found
    hub 3-0:1.0: 2 ports detected
    SCSI subsystem initialized
    libata version 3.00 loaded.
    ata_piix 0000:00:1f.2: version 2.12
    ACPI: PCI Interrupt 0000:00:1f.2[A] -> GSI 18 (level, low) -> IRQ 177
    ata_piix 0000:00:1f.2: MAP [ P0 -- P1 -- ]
    PCI: Setting latency timer of device 0000:00:1f.2 to 64
    scsi0 : ata_piix
    scsi1 : ata_piix
    ata1: SATA max UDMA/133 cmd 0xccb8 ctl 0xccb0 bmdma 0xcc80 irq 177
    ata2: SATA max UDMA/133 cmd 0xcca0 ctl 0xcc98 bmdma 0xcc88 irq 177
    ata1.00: ATA-7: WDC WD800JD-75MSA3, 10.01E04, max UDMA/133
    ata1.00: 156250000 sectors, multi 8: LBA48 NCQ (depth 0/32)
    ata1.00: configured for UDMA/133
    Vendor: ATA Model: WDC WD800JD-75MS Rev: 10.0
    Type: Direct-Access ANSI SCSI revision: 05
    SCSI device sda: 156250000 512-byte hdwr sectors (80000 MB)
    sda: Write Protect is off
    sda: Mode Sense: 00 3a 00 00
    SCSI device sda: drive cache: write back
    SCSI device sda: 156250000 512-byte hdwr sectors (80000 MB)
    sda: Write Protect is off
    sda: Mode Sense: 00 3a 00 00
    SCSI device sda: drive cache: write back
    sda: sda1 sda2
    sd 0:0:0:0: Attached scsi disk sda
    device-mapper: uevent: version 1.0.3
    device-mapper: ioctl: 4.11.5-ioctl (2007-12-12) initialised: dm-devel@redhat.com
    device-mapper: dm-raid45: initialized v0.2594l
    kjournald starting. Commit interval 5 seconds
    EXT3-fs: mounted filesystem with ordered data mode.
    SELinux: Disabled at runtime.
    SELinux: Unregistering netfilter hooks
    type=1404 audit(1308373345.266:2): selinux=0 auid=4294967295 ses=4294967295
    Intel(R) PRO/1000 Network Driver - version 7.3.21-k4-3-NAPI
    Copyright (c) 1999-2006 Intel Corporation.
    GSI 20 sharing vector 0xC9 and IRQ 20
    ACPI: PCI Interrupt 0000:02:04.0[A] -> GSI 32 (level, low) -> IRQ 201
    e1000: 0000:02:04.0: e1000_probe: (PCI:66MHz:32-bit) 00:15:c5:5d:20:3b
    EDAC MC: Ver: 2.0.1 May 31 2011
    input: PC Speaker as /class/input/input0
    Contact your BIOS vendor to see if the E752x error registers can be safely un-hidden
    hda: ATAPI 24X DVD-ROM CD-R/RW drive, 2048kB Cache, UDMA(33)
    Uniform CD-ROM driver Revision: 3.20
    intel_rng: FWH not detected
    e1000: eth0: e1000_probe: Intel(R) PRO/1000 Network Connection
    GSI 21 sharing vector 0xD1 and IRQ 21
    ACPI: PCI Interrupt 0000:04:03.0[A] -> GSI 20 (level, low) -> IRQ 209
    e1000: 0000:04:03.0: e1000_probe: (PCI:33MHz:32-bit) 00:15:c5:5d:20:3c
    sd 0:0:0:0: Attached scsi generic sg0 type 0
    e1000: eth1: e1000_probe: Intel(R) PRO/1000 Network Connection
    shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
    floppy0: no floppy controllers found
    work still pending
    lp: driver loaded but no devices found
    ACPI: Power Button (FF) [PWRF]
    ACPI: Mapper loaded
    dell-wmi: No known WMI GUID found
    ACPI Exception (video-1450): AE_NOT_FOUND, Evaluating _DOD [20060707]
    input: Video Bus as /class/input/input1
    ACPI: Video Device [EVGA] (multi-head: no rom: yes post: no)
    md: Autodetecting RAID arrays.
    md: autorun ...
    md: ... autorun DONE.
    device-mapper: multipath: version 1.0.6 loaded
    EXT3 FS on dm-0, internal journal
    kjournald starting. Commit interval 5 seconds
    EXT3 FS on dm-1, internal journal
    EXT3-fs: mounted filesystem with ordered data mode.
    kjournald starting. Commit interval 5 seconds
    EXT3 FS on sda1, internal journal
    EXT3-fs: mounted filesystem with ordered data mode.
    Adding 3047416k swap on /dev/vg0/swap. Priority:-1 extents:1 across:3047416k
    IA-32 Microcode Update Driver: v1.14a tigran@veritas.com
    Loading iSCSI transport class v2.0-871.
    802.1Q VLAN Support v1.8 Ben Greear greearb@candelatech.com
    All bugs added by David S. Miller davem@redhat.com
    cxgb3i: tag itt 0x1fff, 13 bits, age 0xf, 4 bits.
    iscsi: registered transport (cxgb3i)
    IPv6: Loaded, but administratively disabled, reboot required to enable
    Broadcom NetXtreme II CNIC Driver cnic v2.1.2 (May 26, 2010)
    Broadcom NetXtreme II iSCSI Driver bnx2i v2.1.3 (Aug 10, 2010)
    iscsi: registered transport (bnx2i)
    iscsi: registered transport (tcp)
    iscsi: registered transport (iser)
    iscsi: registered transport (be2iscsi)
    ip_tables: (C) 2000-2006 Netfilter Core Team
    Netfilter messages via NETLINK v0.30.
    ip_conntrack version 2.4 (8192 buckets, 65536 max) - 304 bytes per conntrack
    e1000: eth0 NIC Link is Up 100 Mbps Full Duplex, Flow Control: RX
    ip_set version 4 loaded
    /usr/src/ipset-4.2/kernel/ipt_set.c: checkentry: Cannot find set indentified by id 0 to match
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <8c>
    TDT <cc>
    next_to_use <cc>
    next_to_clean <86>
    buffer_info[next_to_clean]
    time_stamp <100a42580>
    next_to_watch <8f>
    jiffies <100a42e18>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <8c>
    TDT <cc>
    next_to_use <cc>
    next_to_clean <86>
    buffer_info[next_to_clean]
    time_stamp <100a42580>
    next_to_watch <8f>
    jiffies <100a435e8>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <8c>
    TDT <cc>
    next_to_use <cc>
    next_to_clean <86>
    buffer_info[next_to_clean]
    time_stamp <100a42580>
    next_to_watch <8f>
    jiffies <100a43db8>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <8c>
    TDT <cc>
    next_to_use <cc>
    next_to_clean <86>
    buffer_info[next_to_clean]
    time_stamp <100a42580>
    next_to_watch <8f>
    jiffies <100a44588>
    next_to_watch.status <0>
    NETDEV WATCHDOG: eth0: transmit timed out
    e1000: eth0 NIC Link is Up 100 Mbps Full Duplex, Flow Control: RX
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <7d>
    TDT <dc>
    next_to_use <dc>
    next_to_clean <77>
    buffer_info[next_to_clean]
    time_stamp <10967c2fc>
    next_to_watch <80>
    jiffies <10967c95d>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <7d>
    TDT


    next_to_use

    next_to_clean <77>
    buffer_info[next_to_clean]
    time_stamp <10967c2fc>
    next_to_watch <80>
    jiffies <10967d12d>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <7d>
    TDT

    next_to_use

    next_to_clean <77>
    buffer_info[next_to_clean]
    time_stamp <10967c2fc>
    next_to_watch <80>
    jiffies <10967d8fd>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <7d>
    TDT

    next_to_use

    next_to_clean <77>
    buffer_info[next_to_clean]
    time_stamp <10967c2fc>
    next_to_watch <80>
    jiffies <10967e0cd>
    next_to_watch.status <0>
    NETDEV WATCHDOG: eth0: transmit timed out
    e1000: eth0 NIC Link is Up 100 Mbps Full Duplex, Flow Control: RX
    e1000e: Intel(R) PRO/1000 Network Driver - 1.3.17-NAPI
    e1000e: Copyright(c) 1999 - 2011 Intel Corporation.
    e1000e: Intel(R) PRO/1000 Network Driver - 1.3.17-NAPI
    e1000e: Copyright(c) 1999 - 2011 Intel Corporation.
    e1000: eth0 NIC Link is Up 100 Mbps Full Duplex, Flow Control: RX
    ACPI: PCI interrupt for device 0000:04:03.0 disabled
    ACPI: PCI interrupt for device 0000:02:04.0 disabled
    Intel(R) PRO/1000 Network Driver - version 7.3.21-k4-3-NAPI
    Copyright (c) 1999-2006 Intel Corporation.
    PCI: Enabling device 0000:02:04.0 (0110 -> 0113)
    ACPI: PCI Interrupt 0000:02:04.0[A] -> GSI 32 (level, low) -> IRQ 201
    e1000: 0000:02:04.0: e1000_probe: (PCI:66MHz:32-bit) 00:15:c5:5d:20:3b
    e1000: eth0: e1000_probe: Intel(R) PRO/1000 Network Connection
    PCI: Enabling device 0000:04:03.0 (0110 -> 0113)
    ACPI: PCI Interrupt 0000:04:03.0[A] -> GSI 20 (level, low) -> IRQ 209
    e1000: 0000:04:03.0: e1000_probe: (PCI:33MHz:32-bit) 00:15:c5:5d:20:3c
    e1000: eth1: e1000_probe: Intel(R) PRO/1000 Network Connection
    e1000: eth0 NIC Link is Up 100 Mbps Full Duplex, Flow Control: RX
    e1000: eth1 NIC Link is Up 100 Mbps Full Duplex, Flow Control: RX
    ACPI: PCI interrupt for device 0000:04:03.0 disabled
    ACPI: PCI interrupt for device 0000:02:04.0 disabled
    e1000e: Intel(R) PRO/1000 Network Driver - 1.3.17-NAPI
    e1000e: Copyright(c) 1999 - 2011 Intel Corporation.
    Intel(R) PRO/1000 Network Driver - version 7.3.21-k4-3-NAPI
    Copyright (c) 1999-2006 Intel Corporation.
    PCI: Enabling device 0000:02:04.0 (0110 -> 0113)
    ACPI: PCI Interrupt 0000:02:04.0[A] -> GSI 32 (level, low) -> IRQ 201
    e1000: 0000:02:04.0: e1000_probe: (PCI:66MHz:32-bit) 00:15:c5:5d:20:3b
    e1000: eth0: e1000_probe: Intel(R) PRO/1000 Network Connection
    PCI: Enabling device 0000:04:03.0 (0110 -> 0113)
    ACPI: PCI Interrupt 0000:04:03.0[A] -> GSI 20 (level, low) -> IRQ 209
    e1000: 0000:04:03.0: e1000_probe: (PCI:33MHz:32-bit) 00:15:c5:5d:20:3c
    e1000: eth1: e1000_probe: Intel(R) PRO/1000 Network Connection
    e1000: eth0 NIC Link is Up 100 Mbps Full Duplex, Flow Control: RX
    e1000: eth1 NIC Link is Up 100 Mbps Full Duplex, Flow Control: RX
    e1000: eth0 NIC Link is Up 100 Mbps Full Duplex, Flow Control: RX
    ACPI: PCI interrupt for device 0000:04:03.0 disabled
    ACPI: PCI interrupt for device 0000:02:04.0 disabled
    Intel(R) PRO/1000 Network Driver - version 8.0.30-NAPI
    Copyright (c) 1999-2010 Intel Corporation.
    PCI: Enabling device 0000:02:04.0 (0110 -> 0113)
    ACPI: PCI Interrupt 0000:02:04.0[A] -> GSI 32 (level, low) -> IRQ 201
    e1000: 0000:02:04.0: e1000_probe: (PCI:66MHz:32-bit) 00:15:c5:5d:20:3b
    e1000: eth0: e1000_probe: Intel(R) PRO/1000 Network Connection
    PCI: Enabling device 0000:04:03.0 (0110 -> 0113)
    ACPI: PCI Interrupt 0000:04:03.0[A] -> GSI 20 (level, low) -> IRQ 209
    e1000: 0000:04:03.0: e1000_probe: (PCI:33MHz:32-bit) 00:15:c5:5d:20:3c
    e1000: eth1: e1000_probe: Intel(R) PRO/1000 Network Connection
    e1000: eth0: e1000_watchdog_task: NIC Link is Up 100 Mbps Full Duplex, Flow Control: None
    e1000: eth1: e1000_watchdog_task: NIC Link is Up 100 Mbps Full Duplex, Flow Control: None
    e1000: eth0: e1000_watchdog_task: NIC Link is Up 100 Mbps Full Duplex, Flow Control: None
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <40>
    TDT <70>
    next_to_use <70>
    next_to_clean <3a>
    buffer_info[next_to_clean]
    time_stamp <110028131>
    next_to_watch <43>
    jiffies <11002880f>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <40>
    TDT <70>
    next_to_use <70>
    next_to_clean <3a>
    buffer_info[next_to_clean]
    time_stamp <110028131>
    next_to_watch <43>
    jiffies <110028fe0>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <40>
    TDT <70>
    next_to_use <70>
    next_to_clean <3a>
    buffer_info[next_to_clean]
    time_stamp <110028131>
    next_to_watch <43>
    jiffies <1100297af>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <40>
    TDT <70>
    next_to_use <70>
    next_to_clean <3a>
    buffer_info[next_to_clean]
    time_stamp <110028131>
    next_to_watch <43>
    jiffies <110029f7f>
    next_to_watch.status <0>
    NETDEV WATCHDOG: eth0: transmit timed out
    e1000: eth0: e1000_watchdog_task: NIC Link is Up 100 Mbps Full Duplex, Flow Control: None
    /usr/src/ipset-4.2/kernel/ip_set_iphash.c: iphash_retry: rehashing of set tor-list triggered: hashsize grows from 8192 to 12288
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <c>
    TDT <6a>
    next_to_use <6a>
    next_to_clean <8>
    buffer_info[next_to_clean]
    time_stamp <11248f42d>
    next_to_watch <10>
    jiffies <11248fc3f>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <c>
    TDT <6b>
    next_to_use <6b>
    next_to_clean <8>
    buffer_info[next_to_clean]
    time_stamp <11248f42d>
    next_to_watch <10>
    jiffies <11249040e>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <c>
    TDT <6b>
    next_to_use <6b>
    next_to_clean <8>
    buffer_info[next_to_clean]
    time_stamp <11248f42d>
    next_to_watch <10>
    jiffies <112490bde>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <c>
    TDT <6b>
    next_to_use <6b>
    next_to_clean <8>
    buffer_info[next_to_clean]
    time_stamp <11248f42d>
    next_to_watch <10>
    jiffies <1124913ae>
    next_to_watch.status <0>
    NETDEV WATCHDOG: eth0: transmit timed out
    e1000: eth0: e1000_watchdog_task: NIC Link is Up 100 Mbps Full Duplex, Flow Control: None
    /usr/src/ipset-4.2/kernel/ip_set_iphash.c: iphash_retry: rehashing of set tor-list triggered: hashsize grows from 8192 to 12288
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <7b>
    TDT <8>
    next_to_use <8>
    next_to_clean <75>
    buffer_info[next_to_clean]
    time_stamp <11416e27c>
    next_to_watch <84>
    jiffies <11416ece6>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <7b>
    TDT
    next_to_use
    next_to_clean <75>
    buffer_info[next_to_clean]
    time_stamp <11416e27c>
    next_to_watch <84>
    jiffies <11416f4b6>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <7b>
    TDT
    next_to_use
    next_to_clean <75>
    buffer_info[next_to_clean]
    time_stamp <11416e27c>
    next_to_watch <84>
    jiffies <11416fc87>
    next_to_watch.status <0>
    NETDEV WATCHDOG: eth0: transmit timed out
    e1000: eth0: e1000_watchdog_task: NIC Link is Up 100 Mbps Full Duplex, Flow Control: None
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <be>
    TDT <f7>
    next_to_use <f7>
    next_to_clean <b8>
    buffer_info[next_to_clean]
    time_stamp <11462c168>
    next_to_watch <c2>
    jiffies <11462c8c9>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <be>
    TDT <f7>
    next_to_use <f7>
    next_to_clean <b8>
    buffer_info[next_to_clean]
    time_stamp <11462c168>
    next_to_watch <c2>
    jiffies <11462d098>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <be>
    TDT <f7>
    next_to_use <f7>
    next_to_clean <b8>
    buffer_info[next_to_clean]
    time_stamp <11462c168>
    next_to_watch <c2>
    jiffies <11462d868>
    next_to_watch.status <0>
    NETDEV WATCHDOG: eth0: transmit timed out
    e1000: eth0: e1000_watchdog_task: NIC Link is Up 100 Mbps Full Duplex, Flow Control: None
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <ab>
    TDT <fb>
    next_to_use <fb>
    next_to_clean <a7>
    buffer_info[next_to_clean]
    time_stamp <1150e59b3>
    next_to_watch <b0>
    jiffies <1150e6230>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <ab>
    TDT <fb>
    next_to_use <fb>
    next_to_clean <a7>
    buffer_info[next_to_clean]
    time_stamp <1150e59b3>
    next_to_watch <b0>
    jiffies <1150e6a00>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <ab>
    TDT <fb>
    next_to_use <fb>
    next_to_clean <a7>
    buffer_info[next_to_clean]
    time_stamp <1150e59b3>
    next_to_watch <b0>
    jiffies <1150e71d0>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <ab>
    TDT <fb>
    next_to_use <fb>
    next_to_clean <a7>
    buffer_info[next_to_clean]
    time_stamp <1150e59b3>
    next_to_watch <b0>
    jiffies <1150e79a0>
    next_to_watch.status <0>
    NETDEV WATCHDOG: eth0: transmit timed out
    e1000: eth0: e1000_watchdog_task: NIC Link is Up 100 Mbps Full Duplex, Flow Control: None
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <b4>
    TDT <ea>
    next_to_use <ea>
    next_to_clean <ae>
    buffer_info[next_to_clean]
    time_stamp <11514108b>
    next_to_watch <b9>
    jiffies <1151419ab>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <b4>
    TDT <ea>
    next_to_use <ea>
    next_to_clean <ae>
    buffer_info[next_to_clean]
    time_stamp <11514108b>
    next_to_watch <b9>
    jiffies <11514217c>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <b4>
    TDT <ea>
    next_to_use <ea>
    next_to_clean <ae>
    buffer_info[next_to_clean]
    time_stamp <11514108b>
    next_to_watch <b9>
    jiffies <11514294b>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <b4>
    TDT <ea>
    next_to_use <ea>
    next_to_clean <ae>
    buffer_info[next_to_clean]
    time_stamp <11514108b>
    next_to_watch <b9>
    jiffies <11514311b>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <b4>
    TDT <ea>
    next_to_use <ea>
    next_to_clean <ae>
    buffer_info[next_to_clean]
    time_stamp <11514108b>
    next_to_watch <b9>
    jiffies <1151438eb>
    next_to_watch.status <0>
    NETDEV WATCHDOG: eth0: transmit timed out
    e1000: eth0: e1000_watchdog_task: NIC Link is Up 100 Mbps Full Duplex, Flow Control: None
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <4d>
    TDT <9a>
    next_to_use <9a>
    next_to_clean <47>
    buffer_info[next_to_clean]
    time_stamp <11519c476>
    next_to_watch <4e>
    jiffies <11519c9f8>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <4d>
    TDT <9b>
    next_to_use <9b>
    next_to_clean <47>
    buffer_info[next_to_clean]
    time_stamp <11519c476>
    next_to_watch <4e>
    jiffies <11519d1c8>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <4d>
    TDT <9b>
    next_to_use <9b>
    next_to_clean <47>
    buffer_info[next_to_clean]
    time_stamp <11519c476>
    next_to_watch <4e>
    jiffies <11519d998>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <4d>
    TDT <9b>
    next_to_use <9b>
    next_to_clean <47>
    buffer_info[next_to_clean]
    time_stamp <11519c476>
    next_to_watch <4e>
    jiffies <11519e168>
    next_to_watch.status <0>
    NETDEV WATCHDOG: eth0: transmit timed out
    e1000: eth0: e1000_watchdog_task: NIC Link is Up 100 Mbps Full Duplex, Flow Control: None
    ACPI: PCI interrupt for device 0000:04:03.0 disabled
    ACPI: PCI interrupt for device 0000:02:04.0 disabled
    Intel(R) PRO/1000 Network Driver - version 8.0.30-NAPI
    Copyright (c) 1999-2010 Intel Corporation.
    PCI: Enabling device 0000:02:04.0 (0110 -> 0113)
    ACPI: PCI Interrupt 0000:02:04.0[A] -> GSI 32 (level, low) -> IRQ 201
    e1000: 0000:02:04.0: e1000_probe: (PCI:66MHz:32-bit) 00:15:c5:5d:20:3b
    e1000: eth0: e1000_probe: Intel(R) PRO/1000 Network Connection
    PCI: Enabling device 0000:04:03.0 (0110 -> 0113)
    ACPI: PCI Interrupt 0000:04:03.0[A] -> GSI 20 (level, low) -> IRQ 209
    e1000: 0000:04:03.0: e1000_probe: (PCI:33MHz:32-bit) 00:15:c5:5d:20:3c
    e1000: eth1: e1000_probe: Intel(R) PRO/1000 Network Connection
    e1000: eth0: e1000_watchdog_task: NIC Link is Up 100 Mbps Full Duplex, Flow Control: None
    e1000: eth1: e1000_watchdog_task: NIC Link is Up 100 Mbps Full Duplex, Flow Control: None
    e1000: eth0: e1000_watchdog_task: NIC Link is Up 100 Mbps Full Duplex, Flow Control: None
    ACPI: PCI interrupt for device 0000:04:03.0 disabled
    ACPI: PCI interrupt for device 0000:02:04.0 disabled
    Intel(R) PRO/1000 Network Driver - version 8.0.30-NAPI
    Copyright (c) 1999-2010 Intel Corporation.
    PCI: Enabling device 0000:02:04.0 (0110 -> 0113)
    ACPI: PCI Interrupt 0000:02:04.0[A] -> GSI 32 (level, low) -> IRQ 201
    e1000: 0000:02:04.0: e1000_probe: (PCI:66MHz:32-bit) 00:15:c5:5d:20:3b
    e1000: eth0: e1000_probe: Intel(R) PRO/1000 Network Connection
    PCI: Enabling device 0000:04:03.0 (0110 -> 0113)
    ACPI: PCI Interrupt 0000:04:03.0[A] -> GSI 20 (level, low) -> IRQ 209
    e1000: 0000:04:03.0: e1000_probe: (PCI:33MHz:32-bit) 00:15:c5:5d:20:3c
    e1000: eth1: e1000_probe: Intel(R) PRO/1000 Network Connection
    e1000: eth0: e1000_watchdog_task: NIC Link is Up 100 Mbps Full Duplex, Flow Control: None
    e1000: eth1: e1000_watchdog_task: NIC Link is Up 100 Mbps Full Duplex, Flow Control: None
    ACPI: PCI interrupt for device 0000:04:03.0 disabled
    ACPI: PCI interrupt for device 0000:02:04.0 disabled
    Intel(R) PRO/1000 Network Driver - version 8.0.30-NAPI
    Copyright (c) 1999-2010 Intel Corporation.
    PCI: Enabling device 0000:02:04.0 (0110 -> 0113)
    ACPI: PCI Interrupt 0000:02:04.0[A] -> GSI 32 (level, low) -> IRQ 201
    e1000: 0000:02:04.0: e1000_probe: (PCI:66MHz:32-bit) 00:15:c5:5d:20:3b
    e1000: eth0: e1000_probe: Intel(R) PRO/1000 Network Connection
    PCI: Enabling device 0000:04:03.0 (0110 -> 0113)
    ACPI: PCI Interrupt 0000:04:03.0[A] -> GSI 20 (level, low) -> IRQ 209
    e1000: 0000:04:03.0: e1000_probe: (PCI:33MHz:32-bit) 00:15:c5:5d:20:3c
    e1000: eth1: e1000_probe: Intel(R) PRO/1000 Network Connection
    e1000: eth1: e1000_watchdog_task: NIC Link is Up 100 Mbps Full Duplex, Flow Control: None
    e1000: eth0: e1000_watchdog_task: NIC Link is Up 100 Mbps Full Duplex, Flow Control: None
    ACPI: PCI interrupt for device 0000:04:03.0 disabled
    ACPI: PCI interrupt for device 0000:02:04.0 disabled
    e1000: Unknown parameter `ignore_64bit_dma'
    Intel(R) PRO/1000 Network Driver - version 8.0.30-NAPI
    Copyright (c) 1999-2010 Intel Corporation.
    PCI: Enabling device 0000:02:04.0 (0110 -> 0113)
    ACPI: PCI Interrupt 0000:02:04.0[A] -> GSI 32 (level, low) -> IRQ 201
    e1000: 0000:02:04.0: e1000_probe: (PCI:66MHz:32-bit) 00:15:c5:5d:20:3b
    e1000: eth0: e1000_probe: Intel(R) PRO/1000 Network Connection
    PCI: Enabling device 0000:04:03.0 (0110 -> 0113)
    ACPI: PCI Interrupt 0000:04:03.0[A] -> GSI 20 (level, low) -> IRQ 209
    e1000: 0000:04:03.0: e1000_probe: (PCI:33MHz:32-bit) 00:15:c5:5d:20:3c
    e1000: eth1: e1000_probe: Intel(R) PRO/1000 Network Connection
    e1000: eth0: e1000_watchdog_task: NIC Link is Up 100 Mbps Full Duplex, Flow Control: None
    e1000: eth0: e1000_watchdog_task: NIC Link is Up 100 Mbps Full Duplex, Flow Control: None
    e1000: eth1: e1000_watchdog_task: NIC Link is Up 100 Mbps Full Duplex, Flow Control: None
    ACPI: PCI interrupt for device 0000:04:03.0 disabled
    ACPI: PCI interrupt for device 0000:02:04.0 disabled
    Intel(R) PRO/1000 Network Driver - version 8.0.30-NAPI
    Copyright (c) 1999-2010 Intel Corporation.
    PCI: Enabling device 0000:02:04.0 (0110 -> 0113)
    ACPI: PCI Interrupt 0000:02:04.0[A] -> GSI 32 (level, low) -> IRQ 201
    e1000: 0000:02:04.0: e1000_probe: (PCI:66MHz:32-bit) 00:15:c5:5d:20:3b
    e1000: eth0: e1000_probe: Intel(R) PRO/1000 Network Connection
    PCI: Enabling device 0000:04:03.0 (0110 -> 0113)
    ACPI: PCI Interrupt 0000:04:03.0[A] -> GSI 20 (level, low) -> IRQ 209
    e1000: 0000:04:03.0: e1000_probe: (PCI:33MHz:32-bit) 00:15:c5:5d:20:3c
    e1000: eth1: e1000_probe: Intel(R) PRO/1000 Network Connection
    e1000: eth0: e1000_watchdog_task: NIC Link is Up 100 Mbps Full Duplex, Flow Control: None
    e1000: eth1: e1000_watchdog_task: NIC Link is Up 100 Mbps Full Duplex, Flow Control: None
    e1000: eth0: e1000_watchdog_task: NIC Link is Up 100 Mbps Full Duplex, Flow Control: None
    ACPI: PCI interrupt for device 0000:04:03.0 disabled
    ACPI: PCI interrupt for device 0000:02:04.0 disabled
    Intel(R) PRO/1000 Network Driver - version 8.0.30-NAPI
    Copyright (c) 1999-2010 Intel Corporation.
    PCI: Enabling device 0000:02:04.0 (0110 -> 0113)
    ACPI: PCI Interrupt 0000:02:04.0[A] -> GSI 32 (level, low) -> IRQ 201
    e1000: 0000:02:04.0: e1000_probe: (PCI:66MHz:32-bit) 00:15:c5:5d:20:3b
    e1000: eth0: e1000_probe: Intel(R) PRO/1000 Network Connection
    PCI: Enabling device 0000:04:03.0 (0110 -> 0113)
    ACPI: PCI Interrupt 0000:04:03.0[A] -> GSI 20 (level, low) -> IRQ 209
    e1000: 0000:04:03.0: e1000_probe: (PCI:33MHz:32-bit) 00:15:c5:5d:20:3c
    e1000: eth1: e1000_probe: Intel(R) PRO/1000 Network Connection
    e1000: eth1: e1000_watchdog_task: NIC Link is Up 100 Mbps Full Duplex, Flow Control: None
    e1000: eth0: e1000_watchdog_task: NIC Link is Up 100 Mbps Full Duplex, Flow Control: None
    ACPI: PCI interrupt for device 0000:04:03.0 disabled
    ACPI: PCI interrupt for device 0000:02:04.0 disabled
    Intel(R) PRO/1000 Network Driver - version 8.0.30-NAPI
    Copyright (c) 1999-2010 Intel Corporation.
    PCI: Enabling device 0000:02:04.0 (0110 -> 0113)
    ACPI: PCI Interrupt 0000:02:04.0[A] -> GSI 32 (level, low) -> IRQ 201
    e1000: 0000:02:04.0: e1000_probe: (PCI:66MHz:32-bit) 00:15:c5:5d:20:3b
    e1000: eth0: e1000_probe: Intel(R) PRO/1000 Network Connection
    PCI: Enabling device 0000:04:03.0 (0110 -> 0113)
    ACPI: PCI Interrupt 0000:04:03.0[A] -> GSI 20 (level, low) -> IRQ 209
    e1000: 0000:04:03.0: e1000_probe: (PCI:33MHz:32-bit) 00:15:c5:5d:20:3c
    e1000: eth1: e1000_probe: Intel(R) PRO/1000 Network Connection
    e1000: eth0: e1000_watchdog_task: NIC Link is Up 100 Mbps Full Duplex, Flow Control: None
    e1000: eth1: e1000_watchdog_task: NIC Link is Up 100 Mbps Full Duplex, Flow Control: None
    /usr/src/ipset-4.2/kernel/ip_set_iphash.c: iphash_retry: rehashing of set tor-list triggered: hashsize grows from 8192 to 12288
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <13>
    TDT <97>
    next_to_use <97>
    next_to_clean <d>
    buffer_info[next_to_clean]
    time_stamp <118f0d37a>
    next_to_watch <16>
    jiffies <118f0dbcb>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <13>
    TDT <99>
    next_to_use <99>
    next_to_clean <d>
    buffer_info[next_to_clean]
    time_stamp <118f0d37a>
    next_to_watch <16>
    jiffies <118f0e39b>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <13>
    TDT <99>
    next_to_use <99>
    next_to_clean <d>
    buffer_info[next_to_clean]
    time_stamp <118f0d37a>
    next_to_watch <16>
    jiffies <118f0eb6b>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <13>
    TDT <99>
    next_to_use <99>
    next_to_clean <d>
    buffer_info[next_to_clean]
    time_stamp <118f0d37a>
    next_to_watch <16>
    jiffies <118f0f33b>
    next_to_watch.status <0>
    NETDEV WATCHDOG: eth0: transmit timed out
    e1000: eth0: e1000_watchdog_task: NIC Link is Up 100 Mbps Full Duplex, Flow Control: None
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <14>
    TDT <67>
    next_to_use <67>
    next_to_clean <10>
    buffer_info[next_to_clean]
    time_stamp <118f83a2b>
    next_to_watch <19>
    jiffies <118f83f1e>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <14>
    TDT <67>
    next_to_use <67>
    next_to_clean <10>
    buffer_info[next_to_clean]
    time_stamp <118f83a2b>
    next_to_watch <19>
    jiffies <118f846ee>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <14>
    TDT <67>
    next_to_use <67>
    next_to_clean <10>
    buffer_info[next_to_clean]
    time_stamp <118f83a2b>
    next_to_watch <19>
    jiffies <118f84ebe>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <14>
    TDT <67>
    next_to_use <67>
    next_to_clean <10>
    buffer_info[next_to_clean]
    time_stamp <118f83a2b>
    next_to_watch <19>
    jiffies <118f8568e>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <14>
    TDT <67>
    next_to_use <67>
    next_to_clean <10>
    buffer_info[next_to_clean]
    time_stamp <118f83a2b>
    next_to_watch <19>
    jiffies <118f85e5e>
    next_to_watch.status <0>
    NETDEV WATCHDOG: eth0: transmit timed out
    e1000: eth0: e1000_watchdog_task: NIC Link is Up 100 Mbps Full Duplex, Flow Control: None
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <8e>
    TDT <df>
    next_to_use <df>
    next_to_clean <88>
    buffer_info[next_to_clean]
    time_stamp <118fa6dfe>
    next_to_watch <90>
    jiffies <118fa745b>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <8e>
    TDT <e2>
    next_to_use <e2>
    next_to_clean <88>
    buffer_info[next_to_clean]
    time_stamp <118fa6dfe>
    next_to_watch <90>
    jiffies <118fa7c2b>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <8e>
    TDT <e2>
    next_to_use <e2>
    next_to_clean <88>
    buffer_info[next_to_clean]
    time_stamp <118fa6dfe>
    next_to_watch <90>
    jiffies <118fa83fb>
    next_to_watch.status <0>
    NETDEV WATCHDOG: eth0: transmit timed out
    e1000: eth0: e1000_watchdog_task: NIC Link is Up 100 Mbps Full Duplex, Flow Control: None
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <d3>
    TDT <4>
    next_to_use <4>
    next_to_clean <cd>
    buffer_info[next_to_clean]
    time_stamp <11eaec570>
    next_to_watch <d6>
    jiffies <11eaed103>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <d3>
    TDT <4>
    next_to_use <4>
    next_to_clean <cd>
    buffer_info[next_to_clean]
    time_stamp <11eaec570>
    next_to_watch <d6>
    jiffies <11eaed8d4>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <d3>
    TDT <4>
    next_to_use <4>
    next_to_clean <cd>
    buffer_info[next_to_clean]
    time_stamp <11eaec570>
    next_to_watch <d6>
    jiffies <11eaee0a5>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <d3>
    TDT <4>
    next_to_use <4>
    next_to_clean <cd>
    buffer_info[next_to_clean]
    time_stamp <11eaec570>
    next_to_watch <d6>
    jiffies <11eaee876>
    next_to_watch.status <0>
    NETDEV WATCHDOG: eth0: transmit timed out
    e1000: eth0: e1000_watchdog_task: NIC Link is Up 100 Mbps Full Duplex, Flow Control: None
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <1d>
    TDT <47>
    next_to_use <47>
    next_to_clean <17>
    buffer_info[next_to_clean]
    time_stamp <11ee32ef6>
    next_to_watch <1f>
    jiffies <11ee337a5>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <1d>
    TDT <48>
    next_to_use <48>
    next_to_clean <17>
    buffer_info[next_to_clean]
    time_stamp <11ee32ef6>
    next_to_watch <1f>
    jiffies <11ee33f75>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <1d>
    TDT <48>
    next_to_use <48>
    next_to_clean <17>
    buffer_info[next_to_clean]
    time_stamp <11ee32ef6>
    next_to_watch <1f>
    jiffies <11ee34745>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <1d>
    TDT <48>
    next_to_use <48>
    next_to_clean <17>
    buffer_info[next_to_clean]
    time_stamp <11ee32ef6>
    next_to_watch <1f>
    jiffies <11ee34f15>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <1d>
    TDT <48>
    next_to_use <48>
    next_to_clean <17>
    buffer_info[next_to_clean]
    time_stamp <11ee32ef6>
    next_to_watch <1f>
    jiffies <11ee356e5>
    next_to_watch.status <0>
    NETDEV WATCHDOG: eth0: transmit timed out
    e1000: eth0: e1000_watchdog_task: NIC Link is Up 100 Mbps Full Duplex, Flow Control: None
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <6b>
    TDT <b2>
    next_to_use <b2>
    next_to_clean <65>
    buffer_info[next_to_clean]
    time_stamp <122b9b6ec>
    next_to_watch <6d>
    jiffies <122b9bb43>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <6b>
    TDT <b3>
    next_to_use <b3>
    next_to_clean <65>
    buffer_info[next_to_clean]
    time_stamp <122b9b6ec>
    next_to_watch <6d>
    jiffies <122b9c313>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <6b>
    TDT <b3>
    next_to_use <b3>
    next_to_clean <65>
    buffer_info[next_to_clean]
    time_stamp <122b9b6ec>
    next_to_watch <6d>
    jiffies <122b9cae3>
    next_to_watch.status <0>
    NETDEV WATCHDOG: eth0: transmit timed out
    e1000: eth0: e1000_watchdog_task: NIC Link is Up 100 Mbps Full Duplex, Flow Control: None
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <b4>
    TDT <fc>
    next_to_use <fc>
    next_to_clean <b0>
    buffer_info[next_to_clean]
    time_stamp <12459f49b>
    next_to_watch <b8>
    jiffies <12459fc96>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <b4>
    TDT <fd>
    next_to_use <fd>
    next_to_clean <b0>
    buffer_info[next_to_clean]
    time_stamp <12459f49b>
    next_to_watch <b8>
    jiffies <1245a0466>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <b4>
    TDT <fd>
    next_to_use <fd>
    next_to_clean <b0>
    buffer_info[next_to_clean]
    time_stamp <12459f49b>
    next_to_watch <b8>
    jiffies <1245a0c36>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <b4>
    TDT <fd>
    next_to_use <fd>
    next_to_clean <b0>
    buffer_info[next_to_clean]
    time_stamp <12459f49b>
    next_to_watch <b8>
    jiffies <1245a1406>
    next_to_watch.status <0>
    NETDEV WATCHDOG: eth0: transmit timed out
    e1000: eth0: e1000_watchdog_task: NIC Link is Up 100 Mbps Full Duplex, Flow Control: None
    ACPI: PCI interrupt for device 0000:04:03.0 disabled
    ACPI: PCI interrupt for device 0000:02:04.0 disabled
    Intel(R) PRO/1000 Network Driver - version 8.0.30-NAPI
    Copyright (c) 1999-2010 Intel Corporation.
    PCI: Enabling device 0000:02:04.0 (0110 -> 0113)
    ACPI: PCI Interrupt 0000:02:04.0[A] -> GSI 32 (level, low) -> IRQ 201
    e1000: 0000:02:04.0: e1000_validate_option: Transmit Descriptors set to 4096
    e1000: 0000:02:04.0: e1000_validate_option: Receive Descriptors set to 4096
    e1000: 0000:02:04.0: e1000_validate_option: Checksum Offload Disabled
    e1000: 0000:02:04.0: e1000_validate_option: Flow Control Enabled
    e1000: 0000:02:04.0: e1000_validate_option: Transmit Interrupt Delay set to 0
    e1000: 0000:02:04.0: e1000_validate_option: Receive Interrupt Delay set to 0
    e1000: 0000:02:04.0: e1000_check_options: Interrupt Throttling Rate (ints/sec) turned off
    e1000: 0000:02:04.0: e1000_probe: (PCI:66MHz:32-bit) 00:15:c5:5d:20:3b
    e1000: eth0: e1000_probe: Intel(R) PRO/1000 Network Connection
    PCI: Enabling device 0000:04:03.0 (0110 -> 0113)
    ACPI: PCI Interrupt 0000:04:03.0[A] -> GSI 20 (level, low) -> IRQ 209
    e1000: 0000:04:03.0: e1000_probe: (PCI:33MHz:32-bit) 00:15:c5:5d:20:3c
    e1000: eth1: e1000_probe: Intel(R) PRO/1000 Network Connection
    e1000: eth1: e1000_watchdog_task: NIC Link is Up 100 Mbps Full Duplex, Flow Control: None
    e1000: eth0: e1000_watchdog_task: NIC Link is Up 100 Mbps Full Duplex, Flow Control: None
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <47c>
    TDT <4d1>
    next_to_use <4d1>
    next_to_clean <476>
    buffer_info[next_to_clean]
    time_stamp <129c44e23>
    next_to_watch <47e>
    jiffies <129c45989>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <47c>
    TDT <4d1>
    next_to_use <4d1>
    next_to_clean <476>
    buffer_info[next_to_clean]
    time_stamp <129c44e23>
    next_to_watch <47e>
    jiffies <129c46159>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <47c>
    TDT <4d1>
    next_to_use <4d1>
    next_to_clean <476>
    buffer_info[next_to_clean]
    time_stamp <129c44e23>
    next_to_watch <47e>
    jiffies <129c46929>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <47c>
    TDT <4d1>
    next_to_use <4d1>
    next_to_clean <476>
    buffer_info[next_to_clean]
    time_stamp <129c44e23>
    next_to_watch <47e>
    jiffies <129c470f9>
    next_to_watch.status <0>
    NETDEV WATCHDOG: eth0: transmit timed out
    e1000: eth0: e1000_watchdog_task: NIC Link is Up 100 Mbps Full Duplex, Flow Control: None
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <4e0>
    TDT <512>
    next_to_use <512>
    next_to_clean <4da>
    buffer_info[next_to_clean]
    time_stamp <12c12d994>
    next_to_watch <4e5>
    jiffies <12c12e041>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <4e0>
    TDT <517>
    next_to_use <517>
    next_to_clean <4da>
    buffer_info[next_to_clean]
    time_stamp <12c12d994>
    next_to_watch <4e5>
    jiffies <12c12e811>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <4e0>
    TDT <517>
    next_to_use <517>
    next_to_clean <4da>
    buffer_info[next_to_clean]
    time_stamp <12c12d994>
    next_to_watch <4e5>
    jiffies <12c12efe1>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <4e0>
    TDT <517>
    next_to_use <517>
    next_to_clean <4da>
    buffer_info[next_to_clean]
    time_stamp <12c12d994>
    next_to_watch <4e5>
    jiffies <12c12f7b1>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <4e0>
    TDT <517>
    next_to_use <517>
    next_to_clean <4da>
    buffer_info[next_to_clean]
    time_stamp <12c12d994>
    next_to_watch <4e5>
    jiffies <12c12ff81>
    next_to_watch.status <0>
    NETDEV WATCHDOG: eth0: transmit timed out
    e1000: eth0: e1000_watchdog_task: NIC Link is Up 100 Mbps Full Duplex, Flow Control: None
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <c8d>
    TDT <cc7>
    next_to_use <cc7>
    next_to_clean <c87>
    buffer_info[next_to_clean]
    time_stamp <12c176c80>
    next_to_watch <c8e>
    jiffies <12c1775bb>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <c8d>
    TDT <cc8>
    next_to_use <cc8>
    next_to_clean <c87>
    buffer_info[next_to_clean]
    time_stamp <12c176c80>
    next_to_watch <c8e>
    jiffies <12c177d8b>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <c8d>
    TDT <cc8>
    next_to_use <cc8>
    next_to_clean <c87>
    buffer_info[next_to_clean]
    time_stamp <12c176c80>
    next_to_watch <c8e>
    jiffies <12c17855b>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <c8d>
    TDT <cc8>
    next_to_use <cc8>
    next_to_clean <c87>
    buffer_info[next_to_clean]
    time_stamp <12c176c80>
    next_to_watch <c8e>
    jiffies <12c178d2b>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <c8d>
    TDT <cc8>
    next_to_use <cc8>
    next_to_clean <c87>
    buffer_info[next_to_clean]
    time_stamp <12c176c80>
    next_to_watch <c8e>
    jiffies <12c1794fb>
    next_to_watch.status <0>
    NETDEV WATCHDOG: eth0: transmit timed out
    e1000: eth0: e1000_watchdog_task: NIC Link is Up 100 Mbps Full Duplex, Flow Control: None
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <340>
    TDT <3c4>
    next_to_use <3c4>
    next_to_clean <33a>
    buffer_info[next_to_clean]
    time_stamp <12c1933f8>
    next_to_watch <34d>
    jiffies <12c19399f>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <340>
    TDT <3ca>
    next_to_use <3ca>
    next_to_clean <33a>
    buffer_info[next_to_clean]
    time_stamp <12c1933f8>
    next_to_watch <34d>
    jiffies <12c19416f>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <340>
    TDT <3ca>
    next_to_use <3ca>
    next_to_clean <33a>
    buffer_info[next_to_clean]
    time_stamp <12c1933f8>
    next_to_watch <34d>
    jiffies <12c19493f>
    next_to_watch.status <0>
    NETDEV WATCHDOG: eth0: transmit timed out
    e1000: eth0: e1000_watchdog_task: NIC Link is Up 100 Mbps Full Duplex, Flow Control: None
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <ee7>
    TDT <f06>
    next_to_use <f06>
    next_to_clean <ee1>
    buffer_info[next_to_clean]
    time_stamp <12d5c0c00>
    next_to_watch <ee9>
    jiffies <12d5c120f>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <ee7>
    TDT <f06>
    next_to_use <f06>
    next_to_clean <ee1>
    buffer_info[next_to_clean]
    time_stamp <12d5c0c00>
    next_to_watch <ee9>
    jiffies <12d5c19df>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <ee7>
    TDT <f06>
    next_to_use <f06>
    next_to_clean <ee1>
    buffer_info[next_to_clean]
    time_stamp <12d5c0c00>
    next_to_watch <ee9>
    jiffies <12d5c21af>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <ee7>
    TDT <f06>
    next_to_use <f06>
    next_to_clean <ee1>
    buffer_info[next_to_clean]
    time_stamp <12d5c0c00>
    next_to_watch <ee9>
    jiffies <12d5c297f>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <ee7>
    TDT <f06>
    next_to_use <f06>
    next_to_clean <ee1>
    buffer_info[next_to_clean]
    time_stamp <12d5c0c00>
    next_to_watch <ee9>
    jiffies <12d5c314f>
    next_to_watch.status <0>
    NETDEV WATCHDOG: eth0: transmit timed out
    e1000: eth0: e1000_watchdog_task: NIC Link is Up 100 Mbps Full Duplex, Flow Control: None
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <223>
    TDT <24d>
    next_to_use <24d>
    next_to_clean <21d>
    buffer_info[next_to_clean]
    time_stamp <12d965878>
    next_to_watch <224>
    jiffies <12d966195>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <223>
    TDT <24e>
    next_to_use <24e>
    next_to_clean <21d>
    buffer_info[next_to_clean]
    time_stamp <12d965878>
    next_to_watch <224>
    jiffies <12d966965>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <223>
    TDT <24e>
    next_to_use <24e>
    next_to_clean <21d>
    buffer_info[next_to_clean]
    time_stamp <12d965878>
    next_to_watch <224>
    jiffies <12d967135>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <223>
    TDT <24e>
    next_to_use <24e>
    next_to_clean <21d>
    buffer_info[next_to_clean]
    time_stamp <12d965878>
    next_to_watch <224>
    jiffies <12d967905>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <223>
    TDT <24e>
    next_to_use <24e>
    next_to_clean <21d>
    buffer_info[next_to_clean]
    time_stamp <12d965878>
    next_to_watch <224>
    jiffies <12d9680d5>
    next_to_watch.status <0>
    NETDEV WATCHDOG: eth0: transmit timed out
    e1000: eth0: e1000_watchdog_task: NIC Link is Up 100 Mbps Full Duplex, Flow Control: None
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <35c>
    TDT <382>
    next_to_use <382>
    next_to_clean <356>
    buffer_info[next_to_clean]
    time_stamp <12eed8ae3>
    next_to_watch <35e>
    jiffies <12eed9518>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <35c>
    TDT <383>
    next_to_use <383>
    next_to_clean <356>
    buffer_info[next_to_clean]
    time_stamp <12eed8ae3>
    next_to_watch <35e>
    jiffies <12eed9ce8>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <35c>
    TDT <383>
    next_to_use <383>
    next_to_clean <356>
    buffer_info[next_to_clean]
    time_stamp <12eed8ae3>
    next_to_watch <35e>
    jiffies <12eeda4b8>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <35c>
    TDT <383>
    next_to_use <383>
    next_to_clean <356>
    buffer_info[next_to_clean]
    time_stamp <12eed8ae3>
    next_to_watch <35e>
    jiffies <12eedac88>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <35c>
    TDT <383>
    next_to_use <383>
    next_to_clean <356>
    buffer_info[next_to_clean]
    time_stamp <12eed8ae3>
    next_to_watch <35e>
    jiffies <12eedb458>
    next_to_watch.status <0>
    NETDEV WATCHDOG: eth0: transmit timed out
    e1000: eth0: e1000_watchdog_task: NIC Link is Up 100 Mbps Full Duplex, Flow Control: None
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <478>
    TDT <4df>
    next_to_use <4df>
    next_to_clean <472>
    buffer_info[next_to_clean]
    time_stamp <1313d799a>
    next_to_watch <47a>
    jiffies <1313d7fb5>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <478>
    TDT <4e4>
    next_to_use <4e4>
    next_to_clean <472>
    buffer_info[next_to_clean]
    time_stamp <1313d799a>
    next_to_watch <47a>
    jiffies <1313d8785>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <478>
    TDT <4e4>
    next_to_use <4e4>
    next_to_clean <472>
    buffer_info[next_to_clean]
    time_stamp <1313d799a>
    next_to_watch <47a>
    jiffies <1313d8f55>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <478>
    TDT <4e4>
    next_to_use <4e4>
    next_to_clean <472>
    buffer_info[next_to_clean]
    time_stamp <1313d799a>
    next_to_watch <47a>
    jiffies <1313d9725>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <478>
    TDT <4e4>
    next_to_use <4e4>
    next_to_clean <472>
    buffer_info[next_to_clean]
    time_stamp <1313d799a>
    next_to_watch <47a>
    jiffies <1313d9ef5>
    next_to_watch.status <0>
    NETDEV WATCHDOG: eth0: transmit timed out
    e1000: eth0: e1000_watchdog_task: NIC Link is Up 100 Mbps Full Duplex, Flow Control: None
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <fc6>
    TDT <fee>
    next_to_use <fee>
    next_to_clean <fc0>
    buffer_info[next_to_clean]
    time_stamp <1334f1289>
    next_to_watch <fc8>
    jiffies <1334f1869>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <fc6>
    TDT <fef>
    next_to_use <fef>
    next_to_clean <fc0>
    buffer_info[next_to_clean]
    time_stamp <1334f1289>
    next_to_watch <fc8>
    jiffies <1334f2039>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <fc6>
    TDT <fef>
    next_to_use <fef>
    next_to_clean <fc0>
    buffer_info[next_to_clean]
    time_stamp <1334f1289>
    next_to_watch <fc8>
    jiffies <1334f2809>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <fc6>
    TDT <fef>
    next_to_use <fef>
    next_to_clean <fc0>
    buffer_info[next_to_clean]
    time_stamp <1334f1289>
    next_to_watch <fc8>
    jiffies <1334f2fd9>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <fc6>
    TDT <fef>
    next_to_use <fef>
    next_to_clean <fc0>
    buffer_info[next_to_clean]
    time_stamp <1334f1289>
    next_to_watch <fc8>
    jiffies <1334f37a9>
    next_to_watch.status <0>
    NETDEV WATCHDOG: eth0: transmit timed out
    e1000: eth0: e1000_watchdog_task: NIC Link is Up 100 Mbps Full Duplex, Flow Control: None
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <9fe>
    TDT <a2c>
    next_to_use <a2c>
    next_to_clean <9f8>
    buffer_info[next_to_clean]
    time_stamp <134360283>
    next_to_watch <a00>
    jiffies <13436073f>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <9fe>
    TDT <a2d>
    next_to_use <a2d>
    next_to_clean <9f8>
    buffer_info[next_to_clean]
    time_stamp <134360283>
    next_to_watch <a00>
    jiffies <134360f0f>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <9fe>
    TDT <a2d>
    next_to_use <a2d>
    next_to_clean <9f8>
    buffer_info[next_to_clean]
    time_stamp <134360283>
    next_to_watch <a00>
    jiffies <1343616df>
    next_to_watch.status <0>
    NETDEV WATCHDOG: eth0: transmit timed out
    e1000: eth0: e1000_watchdog_task: NIC Link is Up 100 Mbps Full Duplex, Flow Control: None
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <d8e>
    TDT <ddf>
    next_to_use <ddf>
    next_to_clean <d88>
    buffer_info[next_to_clean]
    time_stamp <13634387e>
    next_to_watch <d8f>
    jiffies <136344080>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <d8e>
    TDT <de0>
    next_to_use <de0>
    next_to_clean <d88>
    buffer_info[next_to_clean]
    time_stamp <13634387e>
    next_to_watch <d8f>
    jiffies <136344850>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <d8e>
    TDT <de0>
    next_to_use <de0>
    next_to_clean <d88>
    buffer_info[next_to_clean]
    time_stamp <13634387e>
    next_to_watch <d8f>
    jiffies <136345020>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <d8e>
    TDT <de0>
    next_to_use <de0>
    next_to_clean <d88>
    buffer_info[next_to_clean]
    time_stamp <13634387e>
    next_to_watch <d8f>
    jiffies <1363457f0>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <d8e>
    TDT <de0>
    next_to_use <de0>
    next_to_clean <d88>
    buffer_info[next_to_clean]
    time_stamp <13634387e>
    next_to_watch <d8f>
    jiffies <136345fc0>
    next_to_watch.status <0>
    NETDEV WATCHDOG: eth0: transmit timed out
    e1000: eth0: e1000_watchdog_task: NIC Link is Up 100 Mbps Full Duplex, Flow Control: None
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <cad>
    TDT <d35>
    next_to_use <d35>
    next_to_clean <ca7>
    buffer_info[next_to_clean]
    time_stamp <1365f42ba>
    next_to_watch <cae>
    jiffies <1365f4cc3>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <cad>
    TDT <d35>
    next_to_use <d35>
    next_to_clean <ca7>
    buffer_info[next_to_clean]
    time_stamp <1365f42ba>
    next_to_watch <cae>
    jiffies <1365f5493>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <cad>
    TDT <d35>
    next_to_use <d35>
    next_to_clean <ca7>
    buffer_info[next_to_clean]
    time_stamp <1365f42ba>
    next_to_watch <cae>
    jiffies <1365f5c63>
    next_to_watch.status <0>
    NETDEV WATCHDOG: eth0: transmit timed out
    e1000: eth0: e1000_watchdog_task: NIC Link is Up 100 Mbps Full Duplex, Flow Control: None
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <abe>
    TDT <b40>
    next_to_use <b40>
    next_to_clean <ab8>
    buffer_info[next_to_clean]
    time_stamp <136d158eb>
    next_to_watch <abf>
    jiffies <136d162ec>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <abe>
    TDT <b41>
    next_to_use <b41>
    next_to_clean <ab8>
    buffer_info[next_to_clean]
    time_stamp <136d158eb>
    next_to_watch <abf>
    jiffies <136d16abc>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <abe>
    TDT <b41>
    next_to_use <b41>
    next_to_clean <ab8>
    buffer_info[next_to_clean]
    time_stamp <136d158eb>
    next_to_watch <abf>
    jiffies <136d1728c>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <abe>
    TDT <b41>
    next_to_use <b41>
    next_to_clean <ab8>
    buffer_info[next_to_clean]
    time_stamp <136d158eb>
    next_to_watch <abf>
    jiffies <136d17a5c>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <abe>
    TDT <b41>
    next_to_use <b41>
    next_to_clean <ab8>
    buffer_info[next_to_clean]
    time_stamp <136d158eb>
    next_to_watch <abf>
    jiffies <136d1822c>
    next_to_watch.status <0>
    NETDEV WATCHDOG: eth0: transmit timed out
    e1000: eth0: e1000_watchdog_task: NIC Link is Up 100 Mbps Full Duplex, Flow Control: None
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <edc>
    TDT <f50>
    next_to_use <f50>
    next_to_clean <ed6>
    buffer_info[next_to_clean]
    time_stamp <136ddce40>
    next_to_watch <ee4>
    jiffies <136ddd4e9>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <edc>
    TDT <f53>
    next_to_use <f53>
    next_to_clean <ed6>
    buffer_info[next_to_clean]
    time_stamp <136ddce40>
    next_to_watch <ee4>
    jiffies <136dddcb9>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <edc>
    TDT <f53>
    next_to_use <f53>
    next_to_clean <ed6>
    buffer_info[next_to_clean]
    time_stamp <136ddce40>
    next_to_watch <ee4>
    jiffies <136dde489>
    next_to_watch.status <0>
    NETDEV WATCHDOG: eth0: transmit timed out
    e1000: eth0: e1000_watchdog_task: NIC Link is Up 100 Mbps Full Duplex, Flow Control: None
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH
    TDT <3d>
    next_to_use <3d>
    next_to_clean <5>
    buffer_info[next_to_clean]
    time_stamp <1380dcc53>
    next_to_watch <d>
    jiffies <1380dd111>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH
    TDT <3e>
    next_to_use <3e>
    next_to_clean <5>
    buffer_info[next_to_clean]
    time_stamp <1380dcc53>
    next_to_watch <d>
    jiffies <1380dd8e1>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH
    TDT <3e>
    next_to_use <3e>
    next_to_clean <5>
    buffer_info[next_to_clean]
    time_stamp <1380dcc53>
    next_to_watch <d>
    jiffies <1380de0b1>
    next_to_watch.status <0>
    NETDEV WATCHDOG: eth0: transmit timed out
    e1000: eth0: e1000_watchdog_task: NIC Link is Up 100 Mbps Full Duplex, Flow Control: None
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <95a>
    TDT <a9d>
    next_to_use <a9d>
    next_to_clean <954>
    buffer_info[next_to_clean]
    time_stamp <1394c7fb0>
    next_to_watch <967>
    jiffies <1394c89d4>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <95a>
    TDT <aa1>
    next_to_use <aa1>
    next_to_clean <954>
    buffer_info[next_to_clean]
    time_stamp <1394c7fb0>
    next_to_watch <967>
    jiffies <1394c91a4>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <95a>
    TDT <aa1>
    next_to_use <aa1>
    next_to_clean <954>
    buffer_info[next_to_clean]
    time_stamp <1394c7fb0>
    next_to_watch <967>
    jiffies <1394c9974>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <95a>
    TDT <aa1>
    next_to_use <aa1>
    next_to_clean <954>
    buffer_info[next_to_clean]
    time_stamp <1394c7fb0>
    next_to_watch <967>
    jiffies <1394ca145>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <95a>
    TDT <aa1>
    next_to_use <aa1>
    next_to_clean <954>
    buffer_info[next_to_clean]
    time_stamp <1394c7fb0>
    next_to_watch <967>
    jiffies <1394ca915>
    next_to_watch.status <0>
    NETDEV WATCHDOG: eth0: transmit timed out
    e1000: eth0: e1000_watchdog_task: NIC Link is Up 100 Mbps Full Duplex, Flow Control: None
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <1>
    TDT <52>
    next_to_use <52>
    next_to_clean <ffb>
    buffer_info[next_to_clean]
    time_stamp <13b7cb48a>
    next_to_watch <5>
    jiffies <13b7cbf67>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <1>
    TDT <53>
    next_to_use <53>
    next_to_clean <ffb>
    buffer_info[next_to_clean]
    time_stamp <13b7cb48a>
    next_to_watch <5>
    jiffies <13b7cc737>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <1>
    TDT <53>
    next_to_use <53>
    next_to_clean <ffb>
    buffer_info[next_to_clean]
    time_stamp <13b7cb48a>
    next_to_watch <5>
    jiffies <13b7ccf07>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <1>
    TDT <53>
    next_to_use <53>
    next_to_clean <ffb>
    buffer_info[next_to_clean]
    time_stamp <13b7cb48a>
    next_to_watch <5>
    jiffies <13b7cd6d7>
    next_to_watch.status <0>
    NETDEV WATCHDOG: eth0: transmit timed out
    e1000: eth0: e1000_watchdog_task: NIC Link is Up 100 Mbps Full Duplex, Flow Control: None
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <b6b>
    TDT <bbd>
    next_to_use <bbd>
    next_to_clean <b67>
    buffer_info[next_to_clean]
    time_stamp <13b80c2de>
    next_to_watch <b6d>
    jiffies <13b80cbb8>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <b6b>
    TDT <bbf>
    next_to_use <bbf>
    next_to_clean <b67>
    buffer_info[next_to_clean]
    time_stamp <13b80c2de>
    next_to_watch <b6d>
    jiffies <13b80d388>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <b6b>
    TDT <bbf>
    next_to_use <bbf>
    next_to_clean <b67>
    buffer_info[next_to_clean]
    time_stamp <13b80c2de>
    next_to_watch <b6d>
    jiffies <13b80db58>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <b6b>
    TDT <bbf>
    next_to_use <bbf>
    next_to_clean <b67>
    buffer_info[next_to_clean]
    time_stamp <13b80c2de>
    next_to_watch <b6d>
    jiffies <13b80e328>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <b6b>
    TDT <bbf>
    next_to_use <bbf>
    next_to_clean <b67>
    buffer_info[next_to_clean]
    time_stamp <13b80c2de>
    next_to_watch <b6d>
    jiffies <13b80eaf8>
    next_to_watch.status <0>
    NETDEV WATCHDOG: eth0: transmit timed out
    e1000: eth0: e1000_watchdog_task: NIC Link is Up 100 Mbps Full Duplex, Flow Control: None
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <e99>
    TDT <ee8>
    next_to_use <ee8>
    next_to_clean <e93>
    buffer_info[next_to_clean]
    time_stamp <13bc2329d>
    next_to_watch <e9c>
    jiffies <13bc23ca2>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <e99>
    TDT <ee9>
    next_to_use <ee9>
    next_to_clean <e93>
    buffer_info[next_to_clean]
    time_stamp <13bc2329d>
    next_to_watch <e9c>
    jiffies <13bc24472>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <e99>
    TDT <ee9>
    next_to_use <ee9>
    next_to_clean <e93>
    buffer_info[next_to_clean]
    time_stamp <13bc2329d>
    next_to_watch <e9c>
    jiffies <13bc24c42>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <e99>
    TDT <ee9>
    next_to_use <ee9>
    next_to_clean <e93>
    buffer_info[next_to_clean]
    time_stamp <13bc2329d>
    next_to_watch <e9c>
    jiffies <13bc25412>
    next_to_watch.status <0>
    NETDEV WATCHDOG: eth0: transmit timed out
    e1000: eth0: e1000_watchdog_task: NIC Link is Up 100 Mbps Full Duplex, Flow Control: None
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <861>
    TDT <8c1>
    next_to_use <8c1>
    next_to_clean <85b>
    buffer_info[next_to_clean]
    time_stamp <13d64d332>
    next_to_watch <863>
    jiffies <13d64dc96>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <861>
    TDT <8c2>
    next_to_use <8c2>
    next_to_clean <85b>
    buffer_info[next_to_clean]
    time_stamp <13d64d332>
    next_to_watch <863>
    jiffies <13d64e466>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <861>
    TDT <8c2>
    next_to_use <8c2>
    next_to_clean <85b>
    buffer_info[next_to_clean]
    time_stamp <13d64d332>
    next_to_watch <863>
    jiffies <13d64ec36>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <861>
    TDT <8c2>
    next_to_use <8c2>
    next_to_clean <85b>
    buffer_info[next_to_clean]
    time_stamp <13d64d332>
    next_to_watch <863>
    jiffies <13d64f406>
    next_to_watch.status <0>
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <861>
    TDT <8c2>
    next_to_use <8c2>
    next_to_clean <85b>
    buffer_info[next_to_clean]
    time_stamp <13d64d332>
    next_to_watch <863>
    jiffies <13d64fbd6>
    next_to_watch.status <0>
    NETDEV WATCHDOG: eth0: transmit timed out
    e1000: eth0: e1000_watchdog_task: NIC Link is Up 100 Mbps Full Duplex, Flow Control: None
    e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
    Tx Queue <0>
    TDH <97c>
    TDT <9ca>
    next_to_use <9ca>
    next_to_clean <976>
    buffer_info[next_to_clean]
    time_stamp <13d670576>
    next_to_watch <983>
    jiffies <13d67106f>
    next_to_watch.status <0>
    ACPI: PCI interrupt for device 0000:04:03.0 disabled
    ACPI: PCI interrupt for device 0000:02:04.0 disabled
    Intel(R) PRO/1000 Network Driver - version 8.0.30-NAPI
    Copyright (c) 1999-2010 Intel Corporation.
    PCI: Enabling device 0000:02:04.0 (0110 -> 0113)
    ACPI: PCI Interrupt 0000:02:04.0[A] -> GSI 32 (level, low) -> IRQ 201
    e1000: 0000:02:04.0: e1000_validate_option: Receive Descriptors set to 4096
    e1000: 0000:02:04.0: e1000_check_options: Interrupt Throttling Rate (ints/sec) turned off
    e1000: 0000:02:04.0: e1000_probe: (PCI:66MHz:32-bit) 00:15:c5:5d:20:3b
    e1000: eth0: e1000_probe: Intel(R) PRO/1000 Network Connection
    PCI: Enabling device 0000:04:03.0 (0110 -> 0113)
    ACPI: PCI Interrupt 0000:04:03.0[A] -> GSI 20 (level, low) -> IRQ 209
    e1000: 0000:04:03.0: e1000_probe: (PCI:33MHz:32-bit) 00:15:c5:5d:20:3c
    e1000: eth1: e1000_probe: Intel(R) PRO/1000 Network Connection
    e1000: eth0: e1000_watchdog_task: NIC Link is Up 100 Mbps Full Duplex, Flow Control: None
    e1000: eth1: e1000_watchdog_task: NIC Link is Up 100 Mbps Full Duplex, Flow Control: None

     
  • Tushar Dave

    Tushar Dave - 2011-07-01

    Thanks for the log information.
    Please find the attached debug driver and install it. Next time when the tx hang occurs it will dump the HW ring in dmesg log.
    When you see issue trigger next time, please send full dmesg log in a file and send the ethtool -S eth0 > file_name

     
  • Tushar Dave

    Tushar Dave - 2011-07-01

    e1000-8.0.30_debug driver

     
  • Comment has been marked as spam. 
    Undo

    You can see all pending comments posted by this user  here

    Anonymous - 2011-07-02

    Thanks. I've put the debug drivers in place on both, and reloaded the network modules. I'll update the ticket when I next see the tx hang.

    ethtool -i eth0

    driver: e1000
    version: 8.0.30-NAPI_debug
    firmware-version: N/A
    bus-info: 0000:02:04.0

     
  • Comment has been marked as spam. 
    Undo

    You can see all pending comments posted by this user  here

    Anonymous - 2011-07-02

    Dmesg output

     
    Attachments
  • Comment has been marked as spam. 
    Undo

    You can see all pending comments posted by this user  here

    Anonymous - 2011-07-02

    ethtool -S

     
    Attachments
  • Comment has been marked as spam. 
    Undo

    You can see all pending comments posted by this user  here

    Anonymous - 2011-07-02

    kernel output

     
    Attachments
  • Comment has been marked as spam. 
    Undo

    You can see all pending comments posted by this user  here

    Anonymous - 2011-07-02

    Attached are three files, the ethtool and dmesg file as requested. I've also added the kernel entries from /var/log/messages for the time frame as it looks like a good chunk of the debug data had rolled out of dmesg.

     
  • Comment has been marked as spam. 
    Undo

    You can see all pending comments posted by this user  here

    Anonymous - 2011-07-06

    We've been having a bad day with the servers (several Tx hangs this morning, disrupting users), and whilst I'm very much appreciative for your help so far with this, I've gone ahead and pushed out kernel 2.6.39.2 to one of the servers, and then re-added the 8.0.30 drivers. modprobe settings are as before. We're going to leave it a while before we consider putting the new kernel on the other box, just in case. Here's hoping whatever fixed it for 3.0 (which is still RC, not stable) is in 2.6.39.2!

     
  • Comment has been marked as spam. 
    Undo

    You can see all pending comments posted by this user  here

    Anonymous - 2011-07-06

    Unfortunately that exercise was a little futile. However it did produce some interesting results in dmesg output. We do use iptables on the server for a particular task, but it's not one I consider mission-critical (redirecting users we classify as behaving abnormally). I've shut down iptables "just in case" that happens to stop these problems.

     
  • Comment has been marked as spam. 
    Undo

    You can see all pending comments posted by this user  here

    Anonymous - 2011-07-06

    Dmesg based on 2.9.39 kernel

     
    Attachments
1 2 3 > >> (Page 1 of 3)

Get latest updates about Open Source Projects, Conferences and News.

Sign up for the SourceForge newsletter:





No, thanks