#50 IPMI/BMC stops working when bringing up device

closed
e1000e (109)
5
2015-01-20
2011-09-30
Anonymous
No

As already shortly discussed on the e1000-devel list, here's the summary:

On a Supermicro PDSML-LN2+ (but also found on other boards, I'll add which and what NICs are used) eth0 and the Board management controller share a physical device and mac address albeit having different IP addresses assigned. Using the very same kernel, under Debian Lenny we can query the BMC remotely via ipmitools, but as soon as the iface eth0 is brought up under Debian Squeeze, it stops responding.

Same details:
0d:00.0 Ethernet controller: Intel Corporation 82573E Gigabit Ethernet
Controller (Copper) (rev 03)
0e:00.0 Ethernet controller: Intel Corporation 82573L Gigabit Ethernet
Controller

he 82573E controller is the one shared with the BMC. Under Debian Lenny we
have not encountered any problem with remote access to these systems, e.g.

ipmitool -U USER -P PASSWORD -I lan -H IPMI_IP power status

works. Our setup is that eth0, eth1 and the BMC have distinct IP addresses,
but eth0 and BMC share one MAC, e.g.

n1570:~# ifconfig
eth0 Link encap:Ethernet HWaddr 00:30:48:99:97:3a
inet addr:172.26.15.70 Bcast:172.31.255.255 Mask:255.240.0.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:39289518 errors:0 dropped:0 overruns:0 frame:0
TX packets:28656886 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:100
RX bytes:3059622083 (2.8 GiB) TX bytes:2912754831 (2.7 GiB)
Memory:ee100000-ee120000

eth1 Link encap:Ethernet HWaddr 00:30:48:99:97:3b
inet addr:10.10.15.70 Bcast:10.255.255.255 Mask:255.0.0.0
UP BROADCAST RUNNING MULTICAST MTU:9000 Metric:1
RX packets:733539842 errors:0 dropped:11890044 overruns:0 frame:0
TX packets:373277670 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:3467288935387 (3.1 TiB) TX bytes:350737301109 (326.6 GiB)
Memory:ee200000-ee220000

and
n1570:~# ipmitool lan print
Set in Progress : Set Complete
Auth Type Support : NONE MD2 MD5 PASSWORD
Auth Type Enable : Callback : MD2 MD5 PASSWORD
: User : MD2 MD5 PASSWORD
: Operator : MD2 MD5 PASSWORD
: Admin : MD5 PASSWORD
: OEM : MD2 MD5 PASSWORD
IP Address Source : Static Address
IP Address : 172.27.15.70
Subnet Mask : 255.240.0.0
MAC Address : 00:30:48:99:97:3a
SNMP Community String : public
IP Header : TTL=0x40 Flags=0x40 Precedence=0x00 TOS=0x10
BMC ARP Control : ARP Responses Enabled, Gratuitous ARP Disabled
Gratituous ARP Intrvl : 2.0 seconds
Default Gateway IP : 0.0.0.0
Default Gateway MAC : 00:00:00:00:00:00
Backup Gateway IP : 0.0.0.0
Backup Gateway MAC : 00:00:00:00:00:00
802.1q VLAN ID : Disabled
802.1q VLAN Priority : 0
RMCP+ Cipher Suites : 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14
Cipher Suite Priv Max : Xaaaaaaaaaaaaaa
: X=Cipher Suite Unused
: c=CALLBACK
: u=USER
: o=OPERATOR
: a=ADMIN
: O=OEM

There is no service listening on port 623 used by IPMI thus there should not
be any interference. strace shows some differencing when running ifconfig eth0
up (both attached). From my point of view, the only big difference is

socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) = 4
[...]
ioctl(4, SIOCGIFFLAGS, {ifr_name="eth0", ifr_flags=IFF_BROADCAST|
IFF_MULTICAST}) = 0

with lenny and with squeeze:

socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) = 4
[...]
ioctl(4, SIOCGIFFLAGS, {ifr_name="eth0", ifr_flags=IFF_BROADCAST|
IFF_MULTICAST}) = 0
ioctl(4, SIOCSIFFLAGS, {ifr_name="eth0", ifr_flags=IFF_UP|IFF_BROADCAST|
IFF_RUNNING|IFF_MULTICAST}) = 0

kernel version/modinfo:
lenny:
n1570:/tmp/ethregs-1.13.0# modinfo e1000e
filename: /lib/modules/2.6.32.28-atlas-
generic/kernel/drivers/net/e1000e/e1000e.ko
version: 1.0.2-k2
license: GPL
description: Intel(R) PRO/1000 Network Driver
author: Intel Corporation, linux.nics@intel.com
srcversion: AF3F52EBD9A435E0A141B19
[...]

squeeze:
root@n1670:~# modinfo e1000e
filename: /lib/modules/2.6.32.28-atlas-
generic/kernel/drivers/net/e1000e/e1000e.ko
version: 1.0.2-k2
license: GPL
description: Intel(R) PRO/1000 Network Driver
author: Intel Corporation, linux.nics@intel.com
srcversion: AF3F52EBD9A435E0A141B19

What other information could be helpful for you?

Carsten

Discussion

  • Comment has been marked as spam. 
    Undo

    You can see all pending comments posted by this user  here

    Anonymous - 2011-09-30

    Output of ethregs on Lenny and Squeeze when device was down and up

     
  • Carolyn Wyborny

    Carolyn Wyborny - 2011-10-21

    I'm sorry you're having trouble with our device. Thanks for the ethregs output. Can I also get a full lspci output as well. We will try to see if we can reproduce the problem.

     
  • Andy Cress

    Andy Cress - 2011-10-25

    If the OS and the IPMI BMC share a MAC, it is required that they also are configured for the same IP address. The two can coexist on the same IP because IPMI only uses RMCP port 623.
    If they do not have the same IP, only one of the two will work at the same time, as you have observed.
    You need to fix the IP configuration.

     
  • Carolyn Wyborny

    Carolyn Wyborny - 2011-11-03

    Hello, we are still trying to get a system to repro this problem. This particular combination of features is very rare and actually may have only been produced by SuperMicro. Have you contacted SuperMicro for all updates to BIOS and firmware that are available?

     
  • Comment has been marked as spam. 
    Undo

    You can see all pending comments posted by this user  here

    Anonymous - 2012-01-26

    Hi, sorry for not responding earlier. Via the list I got the correct answer - it all boiled down to setting CrcStripping to 0.

    We had this set on our old machines - albeit in a non-standard place - and thus overlooked this and did not remember settings this after 3+ years.

    Sorry for the hassle, setting
    options e1000e CrcStripping=0
    under /etc/modprobe.d solved this problem!

    Cheers

    Carsten

     

Log in to post a comment.