Recent changes to Netboot_Over_HFI_with_Linux

Netboot_Over_HFI_with_Linux modified by Lissa Valletta

Lissa Valletta — Mon, 23 Jun 2014 17:25:08 -0000

--- v9
+++ v10
@@ -137,6 +137,13 @@
         cp /hfi/scripts/hficonfig /install/postscripts/hficonfig
         chdef c250f07c04ap01 postscripts=servicenode,xcatserver,xcatclient,hficonfig
         updatenode c250f07c04ap01
+
+In **xCAT 2.7**, the xcatserver and xcatclient postscripts are not longer needed in the postscripts table. You should use the following command: 
+
+chdef c250f07c04ap01 postscripts=servicenode,hficonfig 
+
+  
+

   6. Install the DHCP server/client on MN and SN.

Netboot_Over_HFI_with_Linux modified by Bruce

Bruce — Mon, 23 Jun 2014 17:25:07 -0000

--- v8
+++ v9
@@ -1,3 +1,16 @@
+  
+  
+**This document is no longer used! It's content has been integrated into the main xcat cookbooks.**   
+  
+  
+  
+  
+  
+  
+  
+
+
+  
 This doc describes how to set up an xCAT management node and service node to boot compute nodes over the system p HFI network.

 [TOC]

Netboot_Over_HFI_with_Linux modified by Le Yin

Le Yin — Mon, 23 Jun 2014 17:25:06 -0000

--- v7
+++ v8
@@ -155,7 +155,8 @@
         monserver=20.7.4.1
         netboot=yaboot
         nfsserver=20.7.4.1
-        nodetype=lpar,osi
+        nodetype=ppc,osi
+        hwtype=lpar
         os=rhels6
         parent=f07c04fsp1_a
         postscripts=hficonfig

Netboot_Over_HFI_with_Linux modified by

Mon, 23 Jun 2014 17:25:05 -0000

--- v6
+++ v7
@@ -62,12 +62,11 @@
     3. Create soft links and change yaboot.conf to boot from the customized kernel with HFI support

             xdsh c250f07c04ap01 ln -sf /boot/vmlinuz-2.6.32hfi /boot/vmlinuz
-            xdsh c250f07c04ap01 ln -sf /boot/System.map-2.6.32hfi /boot/System.map<pre>
-            
-            Login to the service node and change the "default=" setting in /boot/etc/yaboot.conf to the
-            new label with HFI support.  For example, change it from:
-            
-            <pre>boot=/dev/sda1
+            xdsh c250f07c04ap01 ln -sf /boot/System.map-2.6.32hfi /boot/System.map
+
+Login to the service node and change the "default=" setting in /boot/etc/yaboot.conf to the new label with HFI support. For example, change it from: 
+            
+            boot=/dev/sda1
             init-message="Welcome to Red Hat Enterprise Linux!\nHit <TAB> for boot options"
             partition=3
             timeout=5

Netboot_Over_HFI_with_Linux modified by

Mon, 23 Jun 2014 17:25:04 -0000

--- v5
+++ v6
@@ -64,7 +64,8 @@
             xdsh c250f07c04ap01 ln -sf /boot/vmlinuz-2.6.32hfi /boot/vmlinuz
             xdsh c250f07c04ap01 ln -sf /boot/System.map-2.6.32hfi /boot/System.map<pre>

-            Login to the service node and change the "default=" setting in /boot/etc/yaboot.conf to the new label with HFI support.  For example, change it from:
+            Login to the service node and change the "default=" setting in /boot/etc/yaboot.conf to the
+            new label with HFI support.  For example, change it from:

             <pre>boot=/dev/sda1
             init-message="Welcome to Red Hat Enterprise Linux!\nHit <TAB> for boot options"

Netboot_Over_HFI_with_Linux modified by wang huazhong

wang huazhong — Mon, 23 Jun 2014 17:25:02 -0000

--- v4
+++ v5
@@ -204,7 +204,11 @@

         getmacs c250f07c04ap13 --hfi -D

-Make sure your site.dhcpinterfaces attribute is set correctly to have the MN and SN listen only on the correct NICs. 
+  15. Prepare for the compute node boot 
+        
+        nodeset c250f07c04ap13 netboot
+
+  16. Make sure your site.dhcpinterfaces attribute is set correctly to have the MN and SN listen only on the correct NICs. 

         > tabdump site | grep dhcp
         "dhcpinterfaces","c250mgrs04-pvt|eth0;c250f07c04ap01|hf0",,
@@ -213,15 +217,11 @@

         makedhcp c250f07c04ap13 --HFI

-  15. Prepare for the compute node boot 
-        
-        nodeset c250f07c04ap13 netboot
-
-  16. Reboot the node to boot from network. 
+  17. Reboot the node to boot from network. 

         rnetboot c250f07c04ap13 --hfi

-  17. Open a console to watch the compute node installation. 
+  18. Open a console to watch the compute node installation. 

         rcons c250f07c04ap13

Netboot_Over_HFI_with_Linux modified by Bruce

Bruce — Mon, 23 Jun 2014 17:25:01 -0000

--- v3
+++ v4
@@ -46,7 +46,7 @@
         rpm -ivh --nodeps hfi_util-1.0-0.el6.ppc64.rpm
         rpm -ivh --force net-tools-1.60-102.el6.ppc64.rpm

-  4. Configure the xCAT SN with HFI device driver and new DHCP server/client 
+  4. Configure the xCAT SN with the HFI device driver 
     1. Copy the HFI driver and DHCP packages to service node: 

             xdcp c250f07c04ap01 -R /hfi /hfi
@@ -59,14 +59,12 @@
             xdsh c250f07c04ap01 /sbin/new-kernel-pkg --mkinitrd --depmod --install 2.6.32hfi
             xdsh c250f07c04ap01 /sbin/new-kernel-pkg --rpmposttrans 2.6.32hfi

-    3. Create a softlink and change yaboot.conf to boot from the customized kernel with HFI support 
+    3. Create soft links and change yaboot.conf to boot from the customized kernel with HFI support 

             xdsh c250f07c04ap01 ln -sf /boot/vmlinuz-2.6.32hfi /boot/vmlinuz
             xdsh c250f07c04ap01 ln -sf /boot/System.map-2.6.32hfi /boot/System.map<pre>

-            Login to the service node, change the "default=" setting to the new label with HFI support, for example:
-            
-            Change from:
+            Login to the service node and change the "default=" setting in /boot/etc/yaboot.conf to the new label with HFI support.  For example, change it from:

             <pre>boot=/dev/sda1
             init-message="Welcome to Red Hat Enterprise Linux!\nHit <TAB> for boot options"
@@ -116,31 +114,33 @@
             initrd=/initramfs-2.6.32-71.el6.ppc64.img
             append="rd_NO_LUKS rd_NO_LVM rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrhebsun16 KEYTABLE=us console=hvc0 crashkernel=auto rhgb quiet root=UUID=e2123609-7080-45f0-b583-23d5ef27dbba"

-    4. Reset service nodes to boot from kernel with HFI 
+    4. Reset the service node to boot from the kernel with HFI 

             xdsh c250f07c04ap01 reboot

-  5. Now the HFI interfaces on service nodes should be available on service node, sync the /etc/hosts from MN to SN and configure HFI interfaces with IP address. 
+  5. Now the HFI interfaces should be available on the service node. Sync the /etc/hosts from MN to SN and run the hficonfig postscript to configure the HFI interfaces with IP addresses. 

         xdcp c250f07c04ap01 /etc/hosts /etc/hosts
         cp /hfi/scripts/hficonfig /install/postscripts/hficonfig
         chdef c250f07c04ap01 postscripts=servicenode,xcatserver,xcatclient,hficonfig
         updatenode c250f07c04ap01

-  6. Install DHCP server/client on MN and SN. 
+  6. Install the DHCP server/client on MN and SN. 

         rpm -Uvh /hfi/dhcp/dhcp-4.1.1-13.P1.el6.ppc64.rpm
         rpm -Uvh /hfi/dhcp/dhclient-4.1.1-13.P1.el6.ppc64.rpm
         xdsh c250f07c04ap01 rpm -Uvh /hfi/dhcp/dhcp-4.1.1-13.P1.el6.ppc64.rpm
         xdsh c250f07c04ap01 rpm -Uvh /hfi/dhcp/dhclient-4.1.1-13.P1.el6.ppc64.rpm

-  7. Create new networks for HFI interface. Generally we only need to create one new HFI network for installation. 
+  7. Create a new networks definition in the xCAT database for the HFI interface. Generally we only need to create one new HFI network definition (for hf0) for diskless booting of compute nodes over HFI. 

         mkdef -t network -o hfinet net=20.0.0.0 mask=255.0.0.0 gateway=20.255.255.254 mgtifname=hf0 dhcpserver=20.7.4.1 tftpserver=20.7.4.1 nameservers=20.7.4.1

-  8. Create compute node definitions. Make sure the servicenode and xcatmaster are setting correctly. 
-
-    servicenode attribute should be set to the ethernet IP address or hostname on service node, and xcatmaster should be set to HFI IP address on service node. An example [mkdef stanza file](http://xcat.sourceforge.net/man5/xcatstanzafile.5.html): 
+where the above net, mask, and gateway are appropriate for your hf0 network, and 20.7.4.1 should be changed to be the IP address of the hf0 NIC on your SN. 
+
+  8. Create compute node definitions. Make sure the servicenode and xcatmaster attributes are set correctly. 
+
+    servicenode attribute should be set to the IP address or hostname of the ethernet NIC on the service node that faces the MN. The xcatmaster attribute should be set to hf0 IP address on the service node. An example [mkdef stanza file](http://xcat.sourceforge.net/man5/xcatstanzafile.5.html) for defining the compute node: 

         c250f07c04ap13:
         objtype=node
@@ -167,15 +167,15 @@
         tftpserver=20.7.4.1
         xcatmaster=20.7.4.1

-  9. Create diskless images 
+  9. Create a diskless image for the compute node 

         copycds -n rhels6 /iso/RHEL6.0-20100922.1-Server-ppc64-DVD1.iso

-Where rhels6 stands for the OS version. 
-
-  10. The HFI kernel can be installed by xCAT automatically, but another two packages hfi_util and nettools required rpm --force or --nodeps options, xCAT cannot handle the options automatically. So we need to modify the postinstall file manually which will be run during diskless image generation. 
-
-    Add following lines to /opt/xcat/share/xcat/netboot/rh/compute.rhels6.ppc64.postinstall. (rhels6 stands for the OS version, it should be the same as name in step 8.) 
+Where rhels6 stands for the OS version (RHEL 6 Server). 
+
+  10. The HFI kernel can be installed by xCAT automatically, but the other two packages, hfi_util and nettools, require the rpm options --nodeps and --force respectively, which xCAT cannot handle automatically. So we need to modify the postinstall file manually which will be run during the diskless image generation. 
+
+    Add the following lines to /opt/xcat/share/xcat/netboot/rh/compute.rhels6.ppc64.postinstall. (rhels6 stands for the OS version - it should be the same as the previous step.) 

         cp /hfi/dd/hfi_util-1.0-0.el6.ppc64.rpm /install/netboot/rhels6/ppc64/compute/rootimg/tmp/hfi_util-1.0-0.el6.ppc64.rpm
         cp /hfi/dd/net-tools-1.60-102.el6.ppc64.rpm /install/netboot/rhels6/ppc64/compute/rootimg/tmp/nettools-1.60-102.el6.ppc64.rpm
@@ -186,28 +186,25 @@
         chroot /install/netboot/rhels6/ppc64/compute/rootimg/ /bin/rpm -Uvh /tmp/dhclient-4.1.1-13.P1.el6.ppc64.rpm --force
         chroot /install/netboot/rhels6/ppc64/compute/rootimg/ /bin/rpm -Uvh /tmp/dhcp-4.1.1-13.P1.el6.ppc64.rpm --force

-  
-
-
-  11. Sync the /etc/hosts to diskless image. This is used for the postscript hficonfig to configure all the HFI interfaces on compute nodes. 
+  11. Sync /etc/hosts to the diskless image. This is used by the postscript hficonfig to configure all the HFI interfaces on the compute nodes. 

         cp /hfi/scripts/compute.rhels6.ppc64.synclist /install/custom/netboot/rh/compute.rhels6.ppc64.synclist

-  12. Generate the diskless images 
+  12. Generate the diskless image 

         cd /opt/xcat/share/xcat/netboot/rh/ && /opt/xcat/share/xcat/netboot/rh/genimage -i hf0 -n hf_if -o rhels6 -p compute -k 2.6.32hfi

-where hf0 stands for the boot interface on compute nodes, and hf_hf stands for the boot devices. 
+where hf0 stands for the boot interface on compute nodes, and hf_if stands for the hfi device driver. 

   13. pack the image: 

         packimage -o rhels6 -p compute -a ppc64

-  14. Get the compute node mac address and setup dhcp services. 
+  14. Get the compute node mac address and set up dhcp services. 

         getmacs c250f07c04ap13 --hfi -D

-Make sure your site table have setup dhcp server on MN and SN correctly. 
+Make sure your site.dhcpinterfaces attribute is set correctly to have the MN and SN listen only on the correct NICs. 

         > tabdump site | grep dhcp
         "dhcpinterfaces","c250mgrs04-pvt|eth0;c250f07c04ap01|hf0",,
@@ -216,7 +213,7 @@

         makedhcp c250f07c04ap13 --HFI

-  15. Initialized the boot 
+  15. Prepare for the compute node boot 

         nodeset c250f07c04ap13 netboot

Netboot_Over_HFI_with_Linux modified by Bruce

Bruce — Mon, 23 Jun 2014 17:25:00 -0000

--- v2
+++ v3
@@ -1,23 +1,34 @@
+This doc describes how to set up an xCAT management node and service node to boot compute nodes over the system p HFI network. 
+
+[TOC]
+
 ## On RHEL6:

 ### Preparation:

-  1. We are using following nodes as an example in all the steps: 
-        
-        MN: c250mgrs04-pvt
-        SN: c250f07c04ap01
-        CN: c250f07c04ap13
-
-  2. All the steps are running on xCAT management node. For those commands running on server node, we are using xdsh to the service nodes to run the commands parallized. 
-  3. We assume all the HFI device driver and DHCP server/client packages has been extrated to /hfi directory on xCAT MN. 
+  1. We are using the following node names as an example in this document: 
+        
+        Management node (MN): c250mgrs04-pvt
+        Service node (SN): c250f07c04ap01
+        Compute node (CN): c250f07c04ap13
+
+  2. All the steps should be run on the xCAT management node. For those commands that need to run on the service node, we are using xdsh to run the command remotely on the service node. 
+  3. The following RPMs and scripts referred to in this document currently must be obtained from IBM and put on the xCAT MN in the suggested directories: 
+    * /hfi/dd/kernel-2.6.*hfi-*.ppc64.rpm 
+    * /hfi/dd/hfi_util-*.el6.ppc64.rpm 
+    * /hfi/dd/net-tools-*.el6.ppc64.rpm 
+    * /hfi/dhcp/dhcp-*.el6.ppc64.rpm 
+    * /hfi/dhcp/dhclient-*.el6.ppc64.rpm 
+    * /hfi/scripts/hficonfig 
+    * /hfi/scripts/compute.rhel6.ppc64.synclist 

 ### Steps:

-  1. After installing xCAT, increase the /boot file system size for service node since we need to use a customized kernel on service node, there will be two kernels existing on service nodes, /boot needs more space. 
-
-    This is just for workarounds of customized kernel. After the kernel is accepted by Linux community, there will be only one kernel existing on service node, and we don't need to increase the /boot space at that time. 
-
-    Change file /opt/xcat/share/xcat/install/rh/service.rhels6.ppc64.tmpl, search the line: 
+  1. After installing xCAT on the management node according [XCAT_pLinux_Clusters], increase the /boot file system size for service node since we need to use a customized kernel on service node, there will be two kernels existing on service nodes, /boot needs more space. 
+
+    This is just for a workaround of a customized kernel. After the kernel is accepted by Linux community, there will be only one kernel existing on service node, and we don't need to increase the /boot space at that time. 
+
+    In file /opt/xcat/share/xcat/install/rh/service.rhels6.ppc64.tmpl, change the line: 

     From: 

@@ -27,19 +38,20 @@

         part /boot --size 200 --fstype ext4

-  2. Follow the xCAT MN ans SN setup document to setup xCAT MN and install service nodes. xCAT2pLinux.pdf and xCAT2SetupHierarchy.pdf 
-  3. Install the HFI device drivers on xCAT MN 
-        
+  2. Refer to the rest of [XCAT_pLinux_Clusters] and [Setting_Up_a_Linux_Hierarchical_Cluster] for instructions for setting up a basic xCAT MN and SN and stateless compute nodes. The rest of the instructions below are additional steps you need to take to be able to boot the compute nodes over the HFI network. Eventually, this document will be integrated with the other two. 
+  3. After the basic MN is set up and the SN is installed, install the HFI device drivers on xCAT MN 
+        
+        cd /hfi/dd
         rpm -ivh kernel-2.6.32hfi-3.ppc64.rpm
         rpm -ivh --nodeps hfi_util-1.0-0.el6.ppc64.rpm
         rpm -ivh --force net-tools-1.60-102.el6.ppc64.rpm

-  4. Configure xCAT SN with HFI device driver and new DHCP server/client 
-    1. Create folders on service node and copy the HFI driver and DHCP packages to service node: 
+  4. Configure the xCAT SN with HFI device driver and new DHCP server/client 
+    1. Copy the HFI driver and DHCP packages to service node: 

             xdcp c250f07c04ap01 -R /hfi /hfi

-    2. Install the HFI device drivers and DHCP server for xCAT SN 
+    2. Install the HFI device drivers on the xCAT SN 

             xdsh c250f07c04ap01 rpm -ivh /hfi/dd/kernel-2.6.32hfi-2.ppc64.rpm
             xdsh c250f07c04ap01 rpm -ivh --nodeps /hfi/dd/hfi_util-1.0-0.el6.ppc64.rpm
@@ -47,12 +59,12 @@
             xdsh c250f07c04ap01 /sbin/new-kernel-pkg --mkinitrd --depmod --install 2.6.32hfi
             xdsh c250f07c04ap01 /sbin/new-kernel-pkg --rpmposttrans 2.6.32hfi

-    3. Create softlink and change yaboot.conf to boot from the customized kernel with HFI support 
+    3. Create a softlink and change yaboot.conf to boot from the customized kernel with HFI support 

             xdsh c250f07c04ap01 ln -sf /boot/vmlinuz-2.6.32hfi /boot/vmlinuz
             xdsh c250f07c04ap01 ln -sf /boot/System.map-2.6.32hfi /boot/System.map<pre>

-            Login to service node, change the "default=" setting to the new label with HFI support, for example:
+            Login to the service node, change the "default=" setting to the new label with HFI support, for example:

             Change from:

@@ -215,3 +227,7 @@
   17. Open a console to watch the compute node installation. 

         rcons c250f07c04ap13
+
+## SLES 11
+
+Instructions for SLES 11 will come at a later time.

Netboot_Over_HFI_with_Linux modified by Bruce

Bruce — Mon, 23 Jun 2014 17:24:59 -0000

--- v1
+++ v2
@@ -108,173 +108,110 @@

             xdsh c250f07c04ap01 reboot

+  5. Now the HFI interfaces on service nodes should be available on service node, sync the /etc/hosts from MN to SN and configure HFI interfaces with IP address. 
+        
+        xdcp c250f07c04ap01 /etc/hosts /etc/hosts
+        cp /hfi/scripts/hficonfig /install/postscripts/hficonfig
+        chdef c250f07c04ap01 postscripts=servicenode,xcatserver,xcatclient,hficonfig
+        updatenode c250f07c04ap01
+
+  6. Install DHCP server/client on MN and SN. 
+        
+        rpm -Uvh /hfi/dhcp/dhcp-4.1.1-13.P1.el6.ppc64.rpm
+        rpm -Uvh /hfi/dhcp/dhclient-4.1.1-13.P1.el6.ppc64.rpm
+        xdsh c250f07c04ap01 rpm -Uvh /hfi/dhcp/dhcp-4.1.1-13.P1.el6.ppc64.rpm
+        xdsh c250f07c04ap01 rpm -Uvh /hfi/dhcp/dhclient-4.1.1-13.P1.el6.ppc64.rpm
+
+  7. Create new networks for HFI interface. Generally we only need to create one new HFI network for installation. 
+        
+        mkdef -t network -o hfinet net=20.0.0.0 mask=255.0.0.0 gateway=20.255.255.254 mgtifname=hf0 dhcpserver=20.7.4.1 tftpserver=20.7.4.1 nameservers=20.7.4.1
+
+  8. Create compute node definitions. Make sure the servicenode and xcatmaster are setting correctly. 
+
+    servicenode attribute should be set to the ethernet IP address or hostname on service node, and xcatmaster should be set to HFI IP address on service node. An example [mkdef stanza file](http://xcat.sourceforge.net/man5/xcatstanzafile.5.html): 
+        
+        c250f07c04ap13:
+        objtype=node
+        arch=ppc64
+        cons=fsp
+        groups=lpar,all
+        hcp=f07c04fsp1_a
+        id=13
+        installnic=hf0
+        ip=20.7.4.13
+        mgt=fsp
+        monserver=20.7.4.1
+        netboot=yaboot
+        nfsserver=20.7.4.1
+        nodetype=lpar,osi
+        os=rhels6
+        parent=f07c04fsp1_a
+        postscripts=hficonfig
+        pprofile=c250f07c04ap13
+        primarynic=hf0
+        profile=compute
+        provmethod=netboot
+        servicenode=10.7.4.1
+        tftpserver=20.7.4.1
+        xcatmaster=20.7.4.1
+
+  9. Create diskless images 
+        
+        copycds -n rhels6 /iso/RHEL6.0-20100922.1-Server-ppc64-DVD1.iso
+
+Where rhels6 stands for the OS version. 
+
+  10. The HFI kernel can be installed by xCAT automatically, but another two packages hfi_util and nettools required rpm --force or --nodeps options, xCAT cannot handle the options automatically. So we need to modify the postinstall file manually which will be run during diskless image generation. 
+
+    Add following lines to /opt/xcat/share/xcat/netboot/rh/compute.rhels6.ppc64.postinstall. (rhels6 stands for the OS version, it should be the same as name in step 8.) 
+        
+        cp /hfi/dd/hfi_util-1.0-0.el6.ppc64.rpm /install/netboot/rhels6/ppc64/compute/rootimg/tmp/hfi_util-1.0-0.el6.ppc64.rpm
+        cp /hfi/dd/net-tools-1.60-102.el6.ppc64.rpm /install/netboot/rhels6/ppc64/compute/rootimg/tmp/nettools-1.60-102.el6.ppc64.rpm
+        cp /hfi/dhcp/dhclient-4.1.1-13.P1.el6.ppc64.rpm /install/netboot/rhels6/ppc64/compute/rootimg/tmp/dhclient-.1.1-13.P1.el6.ppc64.rpm
+        cp /hfi/dhcp/dhcp-4.1.1-13.P1.el6.ppc64.rpm /install/netboot/rhels6/ppc64/compute/rootimg/tmp/dhcp-4.1.1-13.P1.el6.ppc64.rpm
+        chroot /install/netboot/rhels6/ppc64/compute/rootimg/ /bin/rpm -ivh /tmp/net-tools-1.60-102.el6.ppc64.rpm --force
+        chroot /install/netboot/rhels6/ppc64/compute/rootimg/ /bin/rpm -ivh /tmp/hfi_util-1.0-0.el6.ppc64.rpm --nodeps --force
+        chroot /install/netboot/rhels6/ppc64/compute/rootimg/ /bin/rpm -Uvh /tmp/dhclient-4.1.1-13.P1.el6.ppc64.rpm --force
+        chroot /install/netboot/rhels6/ppc64/compute/rootimg/ /bin/rpm -Uvh /tmp/dhcp-4.1.1-13.P1.el6.ppc64.rpm --force
+

-5\. Now the HFI interfaces on service nodes should be available on service node, sync the /etc/hosts 
-
-from MN to SN and configure HFI interfaces with IP address. 
-
-xdcp c250f07c04ap01 /etc/hosts /etc/hosts 
-
-cp /hfi/scripts/hficonfig /install/postscripts/hficonfig 
-
-chdef c250f07c04ap01 postscripts=servicenode,xcatserver,xcatclient,hficonfig 
-
-updatenode c250f07c04ap01 
-
-  
-6\. Install DHCP server/client on MN and SN. 
-
-rpm -Uvh /hfi/dhcp/dhcp-4.1.1-13.P1.el6.ppc64.rpm 
-
-rpm -Uvh /hfi/dhcp/dhclient-4.1.1-13.P1.el6.ppc64.rpm 
-
-xdsh c250f07c04ap01 rpm -Uvh /hfi/dhcp/dhcp-4.1.1-13.P1.el6.ppc64.rpm 
-
-xdsh c250f07c04ap01 rpm -Uvh /hfi/dhcp/dhclient-4.1.1-13.P1.el6.ppc64.rpm 
-
-  
-7\. Create new networks for HFI interface. Generally we only need to create one new HFI network for 
-
-installation. 
-
-mkdef -t network -o hfinet net=20.0.0.0 mask=255.0.0.0 gateway=20.255.255.254 mgtifname=hf0 
-
-dhcpserver=20.7.4.1 tftpserver=20.7.4.1 nameservers=20.7.4.1 
-
-  
-8\. Create compute node definitions. Make sure the servicenode and xcatmaster are setting correctly. 
-
-servicenode attribute should be set to the ethernet IP address or hostname on service node, and 
-
-xcatmaster should be set to HFI IP 
-
-address on service node. 
-
-# <xCAT data object stanza file>
-
-c250f07c04ap13: 
-
-objtype=node 
-
-arch=ppc64 
-
-cons=fsp 
-
-groups=lpar,all 
-
-hcp=f07c04fsp1_a 
-
-id=13 
-
-installnic=hf0 
-
-ip=20.7.4.13 
-
-mgt=fsp 
-
-monserver=20.7.4.1 
-
-netboot=yaboot 
-
-nfsserver=20.7.4.1 
-
-nodetype=lpar,osi 
-
-os=rhels6 
-
-parent=f07c04fsp1_a 
-
-postscripts=hficonfig 
-
-pprofile=c250f07c04ap13 
-
-primarynic=hf0 
-
-profile=compute 
-
-provmethod=netboot 
-
-servicenode=10.7.4.1 
-
-tftpserver=20.7.4.1 
-
-xcatmaster=20.7.4.1 
-
-  
-9\. Create diskless images 
-
-copycds -n rhels6 /iso/RHEL6.0-20100922.1-Server-ppc64-DVD1.iso 
-
-Where rhels6 stands for the OS version. 
-
-  
-10\. The HFI kernel can be installed by xCAT automatically, but another two packages hfi_util and nettools required rpm --force or --nodeps options, xCAT cannot handle the options automatically. So we need to modify the postinstall file manually which will be run during diskless image generation. 
-
-Add following lines to /opt/xcat/share/xcat/netboot/rh/compute.rhels6.ppc64.postinstall 
-
-//rhels6 stands for the OS version, it should be the same as name in step 8. 
-
-cp /hfi/dd/hfi_util-1.0-0.el6.ppc64.rpm /install/netboot/rhels6/ppc64/compute/rootimg/tmp/hfi_util-1.0-0.el6.ppc64.rpm 
-
-cp /hfi/dd/net-tools-1.60-102.el6.ppc64.rpm /install/netboot/rhels6/ppc64/compute/rootimg/tmp/nettools-1.60-102.el6.ppc64.rpm 
-
-cp /hfi/dhcp/dhclient-4.1.1-13.P1.el6.ppc64.rpm /install/netboot/rhels6/ppc64/compute/rootimg/tmp/dhclient-4.1.1-13.P1.el6.ppc64.rpm 
-
-cp /hfi/dhcp/dhcp-4.1.1-13.P1.el6.ppc64.rpm /install/netboot/rhels6/ppc64/compute/rootimg/tmp/dhcp-4.1.1-13.P1.el6.ppc64.rpm 
-
-chroot /install/netboot/rhels6/ppc64/compute/rootimg/ /bin/rpm -ivh /tmp/net-tools-1.60-102.el6.ppc64.rpm --force 
-
-chroot /install/netboot/rhels6/ppc64/compute/rootimg/ /bin/rpm -ivh /tmp/hfi_util-1.0-0.el6.ppc64.rpm --nodeps --force 
-
-chroot /install/netboot/rhels6/ppc64/compute/rootimg/ /bin/rpm -Uvh /tmp/dhclient-4.1.1-13.P1.el6.ppc64.rpm --force 
-
-chroot /install/netboot/rhels6/ppc64/compute/rootimg/ /bin/rpm -Uvh /tmp/dhcp-4.1.1-13.P1.el6.ppc64.rpm --force 
-
-  
-11\. Sync the /etc/hosts to diskless image. This is used for the postscript hficonfig to configure all the HFI interfaces on compute nodes. 
-
-cp /hfi/scripts/compute.rhels6.ppc64.synclist 
-
-/install/custom/netboot/rh/compute.rhels6.ppc64.synclist 
-
-  
-12\. Generate the diskless images 
-
-cd /opt/xcat/share/xcat/netboot/rh/ && /opt/xcat/share/xcat/netboot/rh/genimage -i hf0 -n hf_if -o rhels6 -p compute -k 2.6.32hfi 
+
+
+  11. Sync the /etc/hosts to diskless image. This is used for the postscript hficonfig to configure all the HFI interfaces on compute nodes. 
+        
+        cp /hfi/scripts/compute.rhels6.ppc64.synclist /install/custom/netboot/rh/compute.rhels6.ppc64.synclist
+
+  12. Generate the diskless images 
+        
+        cd /opt/xcat/share/xcat/netboot/rh/ && /opt/xcat/share/xcat/netboot/rh/genimage -i hf0 -n hf_if -o rhels6 -p compute -k 2.6.32hfi

 where hf0 stands for the boot interface on compute nodes, and hf_hf stands for the boot devices. 

-  
-13\. pack the image: 
-
-packimage -o rhels6 -p compute -a ppc64 
-
-  
-14\. Get the compute node mac address and setup dhcp services. 
-
-getmacs c250f07c04ap13 --hfi -D 
-
-//Make sure your site table have setup dhcp server on MN and SN correctly. 
-
-#tabdump site | grep dhcp 
-
-"dhcpinterfaces","c250mgrs04-pvt|eth0;c250f07c04ap01|hf0",, 
-
-//Issue makedhcp to setup dhcp services. Specify --HFI option to identify this is a HFI devices. 
-
-makedhcp c250f07c04ap13 --HFI 
-
-  
-15\. Initialized the boot 
-
-nodeset c250f07c04ap13 netboot 
-
-  
-16\. Reboot the node to boot from network. 
-
-rnetboot c250f07c04ap13 --hfi 
-
-  
-17\. Open a console to watch the compute node installation. 
-
-rcons c250f07c04ap13 
+  13. pack the image: 
+        
+        packimage -o rhels6 -p compute -a ppc64
+
+  14. Get the compute node mac address and setup dhcp services. 
+        
+        getmacs c250f07c04ap13 --hfi -D
+
+Make sure your site table have setup dhcp server on MN and SN correctly. 
+        
+        > tabdump site | grep dhcp
+        "dhcpinterfaces","c250mgrs04-pvt|eth0;c250f07c04ap01|hf0",,
+
+Issue makedhcp to setup dhcp services. Specify --HFI option to identify this is a HFI devices. 
+        
+        makedhcp c250f07c04ap13 --HFI
+
+  15. Initialized the boot 
+        
+        nodeset c250f07c04ap13 netboot
+
+  16. Reboot the node to boot from network. 
+        
+        rnetboot c250f07c04ap13 --hfi
+
+  17. Open a console to watch the compute node installation. 
+        
+        rcons c250f07c04ap13

Netboot_Over_HFI_with_Linux modified by Bruce

Bruce — Mon, 23 Jun 2014 17:24:56 -0000

On RHEL6:

Preparation:

We are using following nodes as an example in all the steps:

MN: c250mgrs04-pvt
SN: c250f07c04ap01
CN: c250f07c04ap13

All the steps are running on xCAT management node. For those commands running on server node, we are using xdsh to the service nodes to run the commands parallized.
We assume all the HFI device driver and DHCP server/client packages has been extrated to /hfi directory on xCAT MN.

Steps:

After installing xCAT, increase the /boot file system size for service node since we need to use a customized kernel on service node, there will be two kernels existing on service nodes, /boot needs more space.

This is just for workarounds of customized kernel. After the kernel is accepted by Linux community, there will be only one kernel existing on service node, and we don't need to increase the /boot space at that time.

Change file /opt/xcat/share/xcat/install/rh/service.rhels6.ppc64.tmpl, search the line:

From:
```
part /boot --size 50 --fstype ext4
```
To:
```
part /boot --size 200 --fstype ext4
```
Follow the xCAT MN ans SN setup document to setup xCAT MN and install service nodes. xCAT2pLinux.pdf and xCAT2SetupHierarchy.pdf

Install the HFI device drivers on xCAT MN

rpm -ivh kernel-2.6.32hfi-3.ppc64.rpm
rpm -ivh --nodeps hfi_util-1.0-0.el6.ppc64.rpm
rpm -ivh --force net-tools-1.60-102.el6.ppc64.rpm

Configure xCAT SN with HFI device driver and new DHCP server/client

Create folders on service node and copy the HFI driver and DHCP packages to service node:
```
xdcp c250f07c04ap01 -R /hfi /hfi
```

Install the HFI device drivers and DHCP server for xCAT SN

xdsh c250f07c04ap01 rpm -ivh /hfi/dd/kernel-2.6.32hfi-2.ppc64.rpm
xdsh c250f07c04ap01 rpm -ivh --nodeps /hfi/dd/hfi_util-1.0-0.el6.ppc64.rpm
xdsh c250f07c04ap01 rpm -ivh --force /hfi/dd/net-tools-1.60-102.el6.ppc64.rpm
xdsh c250f07c04ap01 /sbin/new-kernel-pkg --mkinitrd --depmod --install 2.6.32hfi
xdsh c250f07c04ap01 /sbin/new-kernel-pkg --rpmposttrans 2.6.32hfi

Create softlink and change yaboot.conf to boot from the customized kernel with HFI support

xdsh c250f07c04ap01 ln -sf /boot/vmlinuz-2.6.32hfi /boot/vmlinuz
xdsh c250f07c04ap01 ln -sf /boot/System.map-2.6.32hfi /boot/System.map&lt;pre&gt;

Login to service node, change the "default=" setting to the new label with HFI support, for example:

Change from:

&lt;pre&gt;boot=/dev/sda1
init-message="Welcome to Red Hat Enterprise Linux!\nHit &lt;TAB&gt; for boot options"
partition=3
timeout=5
install=/usr/lib/yaboot/yaboot
delay=5
enablecdboot
enableofboot
enablenetboot
nonvram
fstype=raw
default=linux
image=/vmlinuz-2.6.32hfi
label=2.6.32hfi
read-only
initrd=/initrd-2.6.32hfi.img
append="rd_NO_LUKS rd_NO_LVM rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrhebsun16 KEYTABLE=us console=hvc0 crashkernel=auto rhgb quiet root=UUID=e2123609-7080-45f0-b583-23d5ef27dbba"
image=/vmlinuz-2.6.32-71.el6.ppc64
label=linux
read-only
initrd=/initramfs-2.6.32-71.el6.ppc64.img
append="rd_NO_LUKS rd_NO_LVM rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrhebsun16 KEYTABLE=us console=hvc0 crashkernel=auto rhgb quiet root=UUID=e2123609-7080-45f0-b583-23d5ef27dbba"

TO:

        boot=/dev/sda1
        init-message="Welcome to Red Hat Enterprise Linux!\nHit &lt;TAB&gt; for boot options"
        partition=3
        timeout=5
        install=/usr/lib/yaboot/yaboot
        delay=5
        enablecdboot
        enableofboot
        enablenetboot
        nonvram
        fstype=raw
        default=2.6.32hfi
        image=/vmlinuz-2.6.32hfi
        label=2.6.32hfi
        read-only
        initrd=/initrd-2.6.32hfi.img
        append="rd_NO_LUKS rd_NO_LVM rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrhebsun16 KEYTABLE=us console=hvc0 crashkernel=auto rhgb quiet root=UUID=e2123609-7080-45f0-b583-23d5ef27dbba"
        image=/vmlinuz-2.6.32-71.el6.ppc64
        label=linux
        read-only
        initrd=/initramfs-2.6.32-71.el6.ppc64.img
        append="rd_NO_LUKS rd_NO_LVM rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrhebsun16 KEYTABLE=us console=hvc0 crashkernel=auto rhgb quiet root=UUID=e2123609-7080-45f0-b583-23d5ef27dbba"

4. Reset service nodes to boot from kernel with HFI

        xdsh c250f07c04ap01 reboot

5. Now the HFI interfaces on service nodes should be available on service node, sync the /etc/hosts

from MN to SN and configure HFI interfaces with IP address.

xdcp c250f07c04ap01 /etc/hosts /etc/hosts

cp /hfi/scripts/hficonfig /install/postscripts/hficonfig

chdef c250f07c04ap01 postscripts=servicenode,xcatserver,xcatclient,hficonfig

updatenode c250f07c04ap01

6. Install DHCP server/client on MN and SN.

rpm -Uvh /hfi/dhcp/dhcp-4.1.1-13.P1.el6.ppc64.rpm

rpm -Uvh /hfi/dhcp/dhclient-4.1.1-13.P1.el6.ppc64.rpm

xdsh c250f07c04ap01 rpm -Uvh /hfi/dhcp/dhcp-4.1.1-13.P1.el6.ppc64.rpm

xdsh c250f07c04ap01 rpm -Uvh /hfi/dhcp/dhclient-4.1.1-13.P1.el6.ppc64.rpm

7. Create new networks for HFI interface. Generally we only need to create one new HFI network for

installation.

mkdef -t network -o hfinet net=20.0.0.0 mask=255.0.0.0 gateway=20.255.255.254 mgtifname=hf0

dhcpserver=20.7.4.1 tftpserver=20.7.4.1 nameservers=20.7.4.1

8. Create compute node definitions. Make sure the servicenode and xcatmaster are setting correctly.

servicenode attribute should be set to the ethernet IP address or hostname on service node, and

xcatmaster should be set to HFI IP

address on service node.

<xCAT data object stanza file>

c250f07c04ap13:

objtype=node

arch=ppc64

cons=fsp

groups=lpar,all

hcp=f07c04fsp1_a

id=13

installnic=hf0

ip=20.7.4.13

mgt=fsp

monserver=20.7.4.1

netboot=yaboot

nfsserver=20.7.4.1

nodetype=lpar,osi

os=rhels6

parent=f07c04fsp1_a

postscripts=hficonfig

pprofile=c250f07c04ap13

primarynic=hf0

profile=compute

provmethod=netboot

servicenode=10.7.4.1

tftpserver=20.7.4.1

xcatmaster=20.7.4.1

9. Create diskless images

copycds -n rhels6 /iso/RHEL6.0-20100922.1-Server-ppc64-DVD1.iso

Where rhels6 stands for the OS version.

10. The HFI kernel can be installed by xCAT automatically, but another two packages hfi_util and nettools required rpm --force or --nodeps options, xCAT cannot handle the options automatically. So we need to modify the postinstall file manually which will be run during diskless image generation.

Add following lines to /opt/xcat/share/xcat/netboot/rh/compute.rhels6.ppc64.postinstall

//rhels6 stands for the OS version, it should be the same as name in step 8.

cp /hfi/dd/hfi_util-1.0-0.el6.ppc64.rpm /install/netboot/rhels6/ppc64/compute/rootimg/tmp/hfi_util-1.0-0.el6.ppc64.rpm

cp /hfi/dd/net-tools-1.60-102.el6.ppc64.rpm /install/netboot/rhels6/ppc64/compute/rootimg/tmp/nettools-1.60-102.el6.ppc64.rpm

cp /hfi/dhcp/dhclient-4.1.1-13.P1.el6.ppc64.rpm /install/netboot/rhels6/ppc64/compute/rootimg/tmp/dhclient-4.1.1-13.P1.el6.ppc64.rpm

cp /hfi/dhcp/dhcp-4.1.1-13.P1.el6.ppc64.rpm /install/netboot/rhels6/ppc64/compute/rootimg/tmp/dhcp-4.1.1-13.P1.el6.ppc64.rpm

chroot /install/netboot/rhels6/ppc64/compute/rootimg/ /bin/rpm -ivh /tmp/net-tools-1.60-102.el6.ppc64.rpm --force

chroot /install/netboot/rhels6/ppc64/compute/rootimg/ /bin/rpm -ivh /tmp/hfi_util-1.0-0.el6.ppc64.rpm --nodeps --force

chroot /install/netboot/rhels6/ppc64/compute/rootimg/ /bin/rpm -Uvh /tmp/dhclient-4.1.1-13.P1.el6.ppc64.rpm --force

chroot /install/netboot/rhels6/ppc64/compute/rootimg/ /bin/rpm -Uvh /tmp/dhcp-4.1.1-13.P1.el6.ppc64.rpm --force

11. Sync the /etc/hosts to diskless image. This is used for the postscript hficonfig to configure all the HFI interfaces on compute nodes.

cp /hfi/scripts/compute.rhels6.ppc64.synclist

/install/custom/netboot/rh/compute.rhels6.ppc64.synclist

12. Generate the diskless images

cd /opt/xcat/share/xcat/netboot/rh/ && /opt/xcat/share/xcat/netboot/rh/genimage -i hf0 -n hf_if -o rhels6 -p compute -k 2.6.32hfi

where hf0 stands for the boot interface on compute nodes, and hf_hf stands for the boot devices.

13. pack the image:

packimage -o rhels6 -p compute -a ppc64

14. Get the compute node mac address and setup dhcp services.

getmacs c250f07c04ap13 --hfi -D

//Make sure your site table have setup dhcp server on MN and SN correctly.

tabdump site | grep dhcp

"dhcpinterfaces","c250mgrs04-pvt|eth0;c250f07c04ap01|hf0",,

//Issue makedhcp to setup dhcp services. Specify --HFI option to identify this is a HFI devices.

makedhcp c250f07c04ap13 --HFI

15. Initialized the boot

nodeset c250f07c04ap13 netboot

16. Reboot the node to boot from network.

rnetboot c250f07c04ap13 --hfi

17. Open a console to watch the compute node installation.

rcons c250f07c04ap13