Re: [xcat-user] stateful nodes won't boot after IMM network settings are changed or IMM is put on different network

[Rundall, J. D <ru...@il...>]

Thanks, Jarrod.

# nodels object-data[01-04] nodehm.serialflow
object-data01:
object-data02:
object-data03:
object-data04:

It appears that the reason for this is that I pulled these nodes out of the ‘all’ group which is the only way we are applying settings in the nodehm table. And I would have made this change before the other changes listed in my first post for this thread. When they were in the ‘all’ group previously the value would have been ‘hard’.

How might this be affecting the boot and would you have any recommendations?

Thanks again,

Jake

From: Jarrod Johnson <jjo...@le...>
Reply-To: xCAT Users Mailing list <xca...@li...>
Date: Thursday, July 27, 2017 at 9:45 AM
To: xCAT Users Mailing list <xca...@li...>
Subject: Re: [xcat-user] stateful nodes won't boot after IMM network settings are changed or IMM is put on different network

Nodels <noderange> nodehm.serialflow
?


From: Rundall, Jacob D [mailto:ru...@il...]
Sent: Thursday, July 27, 2017 10:12 AM
To: xCAT Users Mailing list
Subject: [xcat-user] stateful nodes won't boot after IMM network settings are changed or IMM is put on different network

I ran into something that seems strange to me. I was workig with a few Lenovo System x3650 M5 nodes yesterday that were deployed with xCAT. They’re running CentOS 7. In order to do some testing I need to move them over to some different networks:
- remove public/routed network from the OS;
- move the OS to new, unrouted management network (for SSH & deployment), with a different IP address, without any xCAT servers;
- move the IMM to new, unrouted service/IPMI network, with a different IP address, also without any xCAT servers.
Also note that I elected to keep their current stateful OS installation rather than redeploying them.

After I did this with the first node I found that it wouldn’t fully boot anymore. Specifically it attempted to boot from the on-disk OS, proceeded through kernel selection, but then would get stuck with “Probing EDD”… showing. (It likely was proceeding past this but just not showing anything else on the crash cart display, more on that later.)

Through some other experimentation I found that any of the following conditions would prevent the node from booting in the same way:
1) without reconfiguring the IMM’s network settings, unplugging the network cable
2) without reconfiguring the IMM’s network settings, connecting the IMM to a switch port on the new service/IPMI network
3) reconfiguring the IP of the IMM (or resetting to factory then reconfiguring on a new IP)
-- this case applies even if I leave the machine connected to our original, production service/IPMI network
But in any case, if I make sure the IMM’s IP is reverted (or doesn’t change) and connect it back up to our production service/IPMI network, then the node will boot again.

I guess my questions are as follow: Is there something on an xCAT-provisioned node (perhaps specific to our hardware, perhaps generally speaking) that requires the IMM/BMC to be not have its IP and/or network changed in order to complete its boot? Is there some kind of communication between the OS and the IMM/BMC that depends on the network connectivity of the IMM/BMC? Is communication with the xCAT master involved (remember the xCAT master is on the production service/IPMI network but not on the new test service/IPMI network)? And are there any ways around this issue?

A few more details:

A) I did disable EDD probing on one of these machines and found that it actually still got stuck (this time with a flashing cursor), so in the previous cases, “Probing EDD”… was simply the last thing that showed on the (crash cart) screen before it got stuck. I also configured tty0 as console on that node to get some more verbose output about what was occurring. Unfortunately this output didn’t appear to make it into /var/log/messages like it seems to on a successful boot (probably because the machine didn’t get far enough along in the boot process to pass it on). But here is a photo:
https://uofi.box.com/s/dt16qvigbgtp0gbz2t0b26m41huv7wo7
I checked quite a few of the lines that appear here and they seem to show up in /var/log/messages after a successful boot as well, so I haven’t uncovered anything that is indicative of the failure I’m seeing. Perhaps there’s something that’s not showing that is telling but I don’t know what it is.

B) Rebuilding plain CentOS 7 from a USB drive allows the nodes to boot with new IMM network settings, with IMMs on the new service/IPMI network.

Thanks much,

Jake Rundall