#4559 configib is slow and unreliable at boot

[Matt Ezell]

I have a cluster with dual-rail IB. I installed the Mellanox driver and OFED tools. I setup xCAT to configure IPoIB addresses for both interfaces:

root@crest-mgmt1:~# lsdef crest4|grep nic
nichostnamesuffixes.ib0=-ib0
nichostnamesuffixes.ib1=-ib1
nicips.ib0=10.36.227.4
nicips.ib1=10.36.227.131
nicnetworks.ib0=ipoib0
nicnetworks.ib1=ipoib1
nictypes.ib0=Infiniband
nictypes.ib1=infiniband
postbootscripts=otherpkgs,nvidia-persistence-mode,confignics

On boot, the configib postscript usually fails:

Feb 4 17:12:13 crest4 xcat: confignics is called: config install nic:0, remove: 0, iba ports:
Feb 4 17:12:13 crest4 xcat: confignics: executed script: configib for nics: ib0,ib1, ports:
Feb 4 17:12:13 crest4 xcat: Not found the driver dameon: rdma or openibd
Feb 4 17:12:13 crest4 xcat: ready
Feb 4 17:12:13 crest4 xcat: done

Even though the openibd service is working fine:

root@crest4:~# service openibd status
openibd start/running
root@crest4:~# echo $?
0

If I run updatenode from the management node, it reliably works, but it's really slow:

root@crest-mgmt1:~# time updatenode crest4 confignics
crest4: Thu Feb 5 17:29:08 EST 2015 Running postscript: confignics
crest4: confignics on crest4: config install nic:0, remove: 0, iba ports:
crest4: ib0!10.36.227.4
crest4: ib1!10.36.227.131
crest4: confignics on crest4: executed script: configib for nics: ib0,ib1, ports:
crest4: openibd start/running
crest4: openibd start/running
crest4: net.core.wmem_max = 1048576
crest4: net.core.rmem_max = 8388608
crest4: net.ipv4.ipfrag_high_thresh = 8388608
crest4: openibd start/running
crest4: Postscript: confignics exited with code 0
crest4: Running of postscripts has completed.

real 1m56.195s
user 0m0.255s
sys 0m0.038s

It appears to be restarting openibd twice, which is really slow.

• How can I make this postscript work reliably on boot?
• How can I speed up configib?

Thanks.

[Matt Ezell]

Ok, so I added some debugging to configib and found that the openibd service was starting on boot. The configib script runs "service openibd status" and it returned "start/pre-start" and likely a non-zero exit code. I added a postinstall script to disable openibd start on boot, and it seems to start reliably.

It's still slow because it restarts openibd several times. Is it possible to reduce that?

[Guang Cheng Li]

xCAT 2.9.1 stream is being closed, Moving to 2.10.

[huweihua]

Hi Matt, I didn't reproduce your first question "configib is unreliable at boot". it seems you have resolved this problem. that's great. For the second question "reduce openibd restart times", I have modified it in xCAT 2.10. if you need, please upgrade xcat to 2.10 when xcat 2.10 release. thanks.

xcat 2.10
commit e9a9987fecffc08cf58dfdb1d7082fcd63d646ca