Recent changes to XCAT_pLinux_Clusters_775

XCAT_pLinux_Clusters_775 modified by Lissa Valletta

Lissa Valletta — Tue, 12 Aug 2014 12:43:01 -0000

--- v2
+++ v3
@@ -4,7 +4,7 @@

 ## Introduction

-This cookbook provides instructions on how to use xCAT to create and deploy a Linux cluster on IBM power 775 system machines.
+This cookbook provides instructions on how to use xCAT to create and deploy a Linux cluster on IBM power 775 system machines.  For other power hardware use [XCAT_pLinux_Clusters].

 The power system machines have the following characteristics:

XCAT_pLinux_Clusters_775 modified by Lissa Valletta

Lissa Valletta — Tue, 12 Aug 2014 12:32:32 -0000

--- v1
+++ v2
@@ -24,7 +24,7 @@
 Here is a summary of the steps required to set up the cluster and what this document will take you through:

   1. Prepare the management node - doing these things before installing the xCAT software helps the process to go more smoothly.
-  2. Install the xCAT software on the management node.
+  2. Install the xCAT software on the management node (EMS).
   3. Configure some cluster wide information
   4. Define a little bit of information in the xCAT database about the ethernet switches and nodes - this is necessary to direct the node discovery process.
   5. Have xCAT configure and start several network daemons - this is necessary for both node discovery and node installation.
@@ -55,15 +55,7 @@

 #### **(Optional)Setup the DHCP interfaces in site table**

-To set up the site table dhcp interfaces for your system p cluster, identify the correct interfaces that xCAT should listen to on your management node and service nodes:
-
-~~~~
-     chdef -t site dhcpinterfaces='pmanagenode|eth1;service|eth0'
-     makedhcp -n
-     service dhcpd restart
-~~~~
-
-For Power 775 cluster:
+

 ~~~~
      chdef -t site dhcpinterfaces='pmanagenode|eth1;service|hf0'
@@ -71,21 +63,19 @@
      service dhcpd restart
 ~~~~

-## Discover and Define Your System p Hardware
-
-The next steps are to discover your hardware on the nextwork, defined it in the xCAT database, configure xCAT's hardware control, and do the initial definition of the LPARs as the compute nodes. These steps are explained in the following 2 documents. Use the one that applies to your environment:
-
-  * For Power 775 clusters, use [XCAT_Power_775_Hardware_Management]
-  * For PowerLinux cluster, use [XCAT_PowerLinux_Hardware_Management]
-  * For all other system p hardware, use [XCAT_System_p_Hardware_Management]
-
-After performing the steps in one of those documents, return here and continue on in this document.
-
-## Additional Setup on the MN for Power 775 Clusters Only
-
-If you are not setting up a Power 775 cluster, skip this section.
-
-The xCAT Management Node must be configured and running on the DB2 database, before installing the following cluster hardware components.
+## Discover and Define Your System p 775 Hardware
+
+The next steps are to discover your hardware on the nextwork, defined it in the xCAT database, configure xCAT's hardware control, and do the initial definition of the LPARs as the compute nodes. These steps are explained in the following document :
+
+  *  [XCAT_Power_775_Hardware_Management]
+ 
+
+After performing those steps  return here and continue on in this document.
+
+## Additional Management Node (EMS) setup
+
+
+The xCAT Management Node must be configured and running on the DB2 database on P775,  before installing the following cluster hardware components.

 This section describes the additional setup required for the Power 775 support. This includes the setup of the cluster hardware components and the installation of TEAL, ISNM, and LoadLeveler on the xCAT management node. TEAL, ISNM and LoadLeveler have dependencies on each other so all three must be installed.

@@ -542,7 +532,7 @@

 The following section (and its subsections) is the standard xCAT procedure for building and deploying a linux stateless image. Some of the example commands refer to the x86_64 architecture, but the procedure is the same on ppc64. Just replace x86_64 with ppc64. Also, when it comes time to boot the nodes, use rnetboot instead of rpower.

-In addition, if you are building/deploying a stateless image on a **p775** cluster, do these additional things when following the procedure below:
+In addition, to build and deploy  a stateless image on a **p775** cluster, do these additional things when following the procedure below:

   * In the section for installing other packages, add the [powerpc-utils rpm](ftp://linuxpatch.ncsa.uiuc.edu/PERCS/powerpc-utils-1.2.2-18.el6.ppc64.rpm) to the otherpkgs directory and the otherpkglist.
   * In the section for using postinstall files, add the following lines to your postinstall script (the location of the rootimg should be changed to your location):
@@ -568,9 +558,7 @@

-### Use network boot to start the installation for non-p775 nodes
-
-    rnetboot compute
+

 ### Use network boot to start the installation for p775 nodes
@@ -698,8 +686,8 @@
   * Back up the xcat database using xcatsnap, important config files and other system config files for reference and for restore later. Prune some of the larger tables:
   *     * tabprune eventlog -a
     * tabprune auditlog -a
-    * tabprune isnm_perf -a (Power 775 only)
-    * tabprune isnm_perf_sum -a (Power 775 only)
+    * tabprune isnm_perf -a 
+    * tabprune isnm_perf_sum -a 
   * Run xcatsnap ( will capture database, config files) and copy to another host. By default it will create in /tmp/xcatsnap two files, for example:
     * xcatsnap.hpcrhmn.10110922.log
     * xcatsnap.hpcrhmn.10110922.tar.gz

XCAT_pLinux_Clusters_775 modified by Lissa Valletta

Lissa Valletta — Tue, 12 Aug 2014 12:21:57 -0000

Introduction
Prepare the Management Node for xCAT Installation
Install xCAT on the Management Node
Configure xCAT
- (Optional)Setup the DHCP interfaces in site table
Discover and Define Your System p Hardware
Additional Setup on the MN for Power 775 Clusters Only
Setup the xCAT MN for a Hierarchical Cluster
Complete the Definition of the Compute Nodes
Install Stateful Nodes
Stateless Node Deployment
Statelite Node Deployment
Advanced features
- Use the driver update disk:
- Setup Kdump Service over Ethernet/HFI on diskless Linux (for xCAT 2.6 and higher)
References
Appendix A: Migrate your Management Node to a new Service Pack of Linux
Appendix B: Install your Management Node to a new Release of Linux

Introduction

This cookbook provides instructions on how to use xCAT to create and deploy a Linux cluster on IBM power 775 system machines.

The power system machines have the following characteristics:

May have multiple LPARs (an LPAR will be the target machine to install an operating system image on, i.e. the LPAR will be the compute node).
The Ethernet card and SCSI disk can be virtual devices.
An HMC or xCAT DFM (Direct FSP/BPA Management) is used for the HCP (hardware control point) to control them.

xCAT supports three types of installations for compute nodes: Diskfull installation (Statefull), Diskless Stateless, and Diskless Statelite. xCAT also supports hierarchical clusters where one or more service nodes are used to handle the installation and management of compute nodes. (Instructions and references will be given later in this document for setting up a hierarchical cluster.)

This document will guide you through installing xCAT on your management node, configuring your cluster, deploying a Linux operating system to your compute nodes, and optionally upgrading firmware on your power system hardware.

Overview of Cluster Setup Process

Here is a summary of the steps required to set up the cluster and what this document will take you through:

Prepare the management node - doing these things before installing the xCAT software helps the process to go more smoothly.
Install the xCAT software on the management node.
Configure some cluster wide information
Define a little bit of information in the xCAT database about the ethernet switches and nodes - this is necessary to direct the node discovery process.
Have xCAT configure and start several network daemons - this is necessary for both node discovery and node installation.
Discovery the nodes - during this phase, xCAT configures the FSP's and collects many attributes about each node and stores them in the database.
Set up the OS images and install the nodes.

Distro-specific Steps

[RH] indicates that step only needs to be done for RHEL and Red Hat based distros (CentOS, Scientific Linux, and in most cases Fedora).
[SLES] indicates that step only needs to be done for SLES.

Command Man Pages and Database Attribute Descriptions

All of the commands used in this document are described in the xCAT man pages.
All of the database attributes referred to in this document are described in the xCAT database object and table descriptions.

Prepare the Management Node for xCAT Installation

These steps prepare the Management Node for xCAT Installation.

Install the Management Node OS
Supported OS and Hardware
[RH] Ensure that SELinux is Disabled
Disable the Firewall
Set Up the Networks
Configure NICS
Prevent DHCP client from overwriting DNS configuration (Optional)
Configure hostname
Setup basic hosts file
Setup the TimeZone
Create a Separate File system for /install (optional)
Restart Management Node

Install the Management Node OS

Install one of the supported distros on the Management Node (MN). It is recommended to ensure that dhcp, bind (not bind-chroot), httpd, nfs-utils, and perl-XML-Parser are installed. (But if not, the process of installing the xCAT software later will pull them in, assuming you follow the steps to make the distro RPMs available.)

Hardware requirements for your xCAT management node are dependent on your cluster size and configuration. A minimum requirement for an xCAT Management Node or Service Node that is dedicated to running xCAT to install a small cluster ( < 16 nodes) should have 4-6 Gigabytes of memory. A medium size cluster, 6-8 Gigabytes of memory; and a large cluster, 16 Gigabytes or more. Keeping swapping to a minimum should be a goal.

Supported OS and Hardware

For a list of supported OS and Hardware, refer to XCAT_Features.

[RH] Ensure that SELinux is Disabled

'''Note:''' you can skip this step in xCAT 2.8.1 and above, because xCAT does it automatically when it is installed.

To disable SELinux manually:

 echo 0 > /selinux/enforce
 sed -i 's/^SELINUX=.*$/SELINUX=disabled/' /etc/selinux/config

Disable the Firewall

Note: you can skip this step in xCAT 2.8 and above, because xCAT does it automatically when it is installed.

The management node provides many services to the cluster nodes, but the firewall on the management node can interfere with this. If your cluster is on a secure network, the easiest thing to do is to disable the firewall on the Management Mode:

For RH:

 service iptables stop
 chkconfig iptables off

If disabling the firewall completely isn't an option, configure iptables to allow the following services on the NIC that faces the cluster: DHCP, TFTP, NFS, HTTP, DNS.

For SLES:

 SuSEfirewall2 stop

Set Up the Networks

The xCAT installation process will scan and populate certain settings from the running configuration. Having the networks configured ahead of time will aid in correct configuration. (After installation of xCAT, all the networks in the cluster must be defined in the xCAT networks table before starting to install cluster nodes.) When xCAT is installed on the Management Node, it will automatically run makenetworks to create an entry in the networks table for each of the networks the management node is on. Additional network configurations can be added to the xCAT networks table manually later if needed.

The networks that are typically used in a cluster are:

Management network - used by the management node to install and manage the OS of the nodes. The MN and in-band NIC of the nodes are connected to this network. If you have a large cluster with service nodes, sometimes this network is segregated into separate VLANs for each service node. See Setting Up a Linux Hierarchical Cluster for details.
Service network - used by the management node to control the nodes out of band via the BMC. If the BMCs are configured in shared mode, then this network can be combined with the management network.
Application network - used by the HPC applications on the compute nodes. Usually an IB network.
Site (Public) network - used to access the management node and sometimes for the compute nodes to provide services to the site.

In our example, we only deal with the management network because:

the BMCs are in shared mode, so they don't need a separate service network
we are not showing how to have xCAT automatically configure the application network NICs. See Configuring_Secondary_Adapters if you are interested in that.
under normal circumstances there is no need to put the site network in the networks table

For a sample Networks Setup, see the following example: Setting_Up_a_Linux_xCAT_Mgmt_Node#Appendix_A:_Network_Table_Setup_Example

Configure NICS

Configure the cluster facing NIC(s) on the management node.
For example edit the following files:

On RH: /etc/sysconfig/network-scripts/ifcfg-eth1
On SLES: /etc/sysconfig/network/ifcfg-eth1

 DEVICE=eth1
 ONBOOT=yes
 BOOTPROTO=static
 IPADDR=172.20.0.1
 NETMASK=255.240.0.0

Prevent DHCP client from overwriting DNS configuration (Optional)

If the public facing NIC on your management node is configured by DHCP, you may want to set '''PEERDNS=no''' in the NIC's config file to prevent the dhclient from rewriting /etc/resolv.conf. This would be important if you will be configuring DNS on the management node (via makedns - covered later in this doc) and want the management node itself to use that DNS. In this case, set '''PEERDNS=no''' in each /etc/sysconfig/network-scripts/ifcfg-* file that has '''BOOTPROTO=dhcp'''.

On the other hand, if you '''want''' dhclient to configure /etc/resolv.conf on your management node, then don't set PEERDNS=no in the NIC config files.

Configure hostname

The xCAT management node hostname should be configured before installing xCAT on the management node. The hostname or its resolvable ip address will be used as the default master name in the xCAT site table, when installed. This name needs to be the one that will resolve to the cluster-facing NIC. Short hostnames (no domain) are the norm for the management node and all cluster nodes. Node names should never end in "-enx" for any x.

To set the hostname, edit /etc/sysconfig/network to contain, for example:

 HOSTNAME=mgt

If you run hostname command, if should return the same:

 # hostname
 mgt

Setup basic hosts file

Ensure that at least the management node is in /etc/hosts:

 127.0.0.1               localhost.localdomain localhost
 ::1                     localhost6.localdomain6 localhost6
 ###
 172.20.0.1 mgt mgt.cluster

Setup the TimeZone

When using the management node to install compute nodes, the timezone configuration on the management node will be inherited by the compute nodes. So it is recommended to setup the correct timezone on the management node. To do this on RHEL, see http://www.redhat.com/advice/tips/timezone.html. The process is similar, but not identical, for SLES. (Just google it.)

You can also optionally set up the MN as an NTP for the cluster. See Setting_up_NTP_in_xCAT.

Create a Separate File system for /install (optional)

It is not required, but recommended, that you create a separate file system for the /install directory on the Management Node. The size should be at least 30 meg to hold to allow space for several install images.

Restart Management Node

Note: in xCAT 2.8 and above, you do not need to restart the management node. Simply restart the cluster-facing NIC, for example: ifdown eth1; ifup eth1

For xCAT 2.7 and below, though it is possible to restart the correct services for all settings, the simplest step would be to reboot the Management Node at this point.

Install xCAT on the Management Node

Get the xCAT Installation Source

There are two options to get the installation source of xCAT:

download the xCAT installation packages
or install directly from the internet-hosted repository

Pick either one, but not both.

Option 1: Prepare for the Install of xCAT without Internet Access

If not able to, or not want to, use the live internet repository, choose this option.

Go to the Download xCAT site and download the level of xCAT tarball you desire. Go to the xCAT Dependencies Download page and download the latest snap of the xCAT dependency tarball. (The latest snap of the xCAT dependency tarball will work with any version of xCAT.)

Copy the files to the Management Node (MN) and untar them:

mkdir /root/xcat2
cd /root/xcat2
tar jxvf xcat-core-2.*.tar.bz2     # or core-rpms-snap.tar.bz2
tar jxvf xcat-dep-*.tar.bz2

Point yum/zypper to the local repositories for xCAT and its dependencies:

[RH]:

cd /root/xcat2/xcat-dep/<release>/<arch>;
./mklocalrepo.sh
cd /root/xcat2/xcat-core
./mklocalrepo.sh

[SLES 11]:

 zypper ar file:///root/xcat2/xcat-dep/sles11/<arch> xCAT-dep 
 zypper ar file:///root/xcat2/xcat-core  xcat-core

[SLES 10.2+]:

zypper sa file:///root/xcat2/xcat-dep/sles10/<arch> xCAT-dep
zypper sa file:///root/xcat2/xcat-core xcat-core

Option 2: Use the Internet-hosted xCAT Repository

When using the live internet repository, you need to first make sure that name resolution on your management node is at least set up enough to resolve sourceforge.net. Then make sure the correct repo files are in /etc/yum.repos.d.

Internet repo for xCAT-core

You could use the official release or latest snapshot build or development build, based on your requirements.

To get the repo file for the current official release:

[RH]:

wget http://sourceforge.net/projects/xcat/files/yum/<xCAT-release>/xcat-core/xCAT-core.repo

for example:

cd /etc/yum.repos.d
wget http://sourceforge.net/projects/xcat/files/yum/2.8/xcat-core/xCAT-core.repo

[SLES11]:

zypper ar -t rpm-md http://sourceforge.net/projects/xcat/files/yum/<xCAT-release>/xcat-core xCAT-core

for example:

zypper ar -t rpm-md http://sourceforge.net/projects/xcat/files/yum/2.8/xcat-core xCAT-core

[SLES10.2+]:

zypper sa http://sourceforge.net/projects/xcat/files/yum/<xCAT-release\>/xcat-core xCAT-core

for example:

 zypper sa http://sourceforge.net/projects/xcat/files/yum/2.8/xcat-core xCAT-core

To get the repo file for the latest snapshot build, which includes the latest bug fixes, but is not completely tested:

[RH]:

wget http://sourceforge.net/projects/xcat/files/yum/<xCAT-release>/core-snap/xCAT-core.repo

for example:

cd /etc/yum.repos.d
wget http://sourceforge.net/projects/xcat/files/yum/2.8/core-snap/xCAT-core.repo

[SLES11]:

zypper ar -t rpm-md http://sourceforge.net/projects/xcat/files/yum/<xCAT-release>/core-snap xCAT-core

for example:

zypper ar -t rpm-md http://sourceforge.net/projects/xcat/files/yum/2.8/core-snap xCAT-core

[SLES10.2+]:

zypper sa http://sourceforge.net/projects/xcat/files/yum/<xCAT-release\>/core-snap xCAT-core

for example:

 zypper sa http://sourceforge.net/projects/xcat/files/yum/2.8/core-snap xCAT-core

To get the repo file for the latest development build, which is the snap shot build of the new version we are actively developing. This version has not been released yet. Use at your own risk:

[RH]:

wget http://sourceforge.net/projects/xcat/files/yum/devel/core-snap/xCAT-core.repo

[SLES11]:

zypper ar -t rpm-md http://sourceforge.net/projects/xcat/files/yum/devel/core-snap xCAT-core

[SLES10.2+]:

zypper sa http://sourceforge.net/projects/xcat/files/yum/devel/core-snap xCAT-core

Internet repo for xCAT-dep

To get the repo file for xCAT-dep packages:

[RH]:

wget http://sourceforge.net/projects/xcat/files/yum/xcat-dep/<OS-release>/<arch>/xCAT-dep.repo

for example:

wget http://sourceforge.net/projects/xcat/files/yum/xcat-dep/rh6/x86_64/xCAT-dep.repo

[SLES11]:

zypper ar -t rpm-md http://sourceforge.net/projects/xcat/files/yum/xcat-dep/<OS-release>/<arch> xCAT-dep

for example:

zypper ar -t rpm-md http://sourceforge.net/projects/xcat/files/yum/xcat-dep/sles11/x86_64 xCAT-dep

[SLES10.2+]:

zypper sa http://sourceforge.net/projects/xcat/files/yum/xcat-dep/<OS-release\>/<arch\> xCAT-dep

for example:

zypper sa http://sourceforge.net/projects/xcat/files/yum/xcat-dep/sles10/x86_64 xCAT-dep

For both Options: Make Required Packages From the Distro Available

xCAT uses on several packages that come from the Linux distro. Follow this section to create the repository of the OS on the Management Node.

See the following documentation:

Setting Up the OS Repository on the Mgmt Node

Install xCAT Packages

[RH]: Use yum to install xCAT and all the dependencies:

yum clean metadata
yum install xCAT

[SLES]Use zypper to install xCAT and all the dependencies:

zypper install xCAT

(Optional) Install the Packages for sysclone

Note: in xCAT 2.8.2 and above, xCAT supports cloning new nodes from a pre-installed/pre-configured node, we call this provisioning method as sysclone. It leverages the opensource tool systemimager. If you will be installing stateful(diskful) nodes using the sysclone provmethod, you need to install systemimager and all the dependencies:

[RH]: Use yum to install systemimager and all the dependencies:

yum install systemimager-server

[SLES]: Use zypper to install systemimager and all the dependencies:

zypper install systemimager-server

Quick Test of xCAT Installation

Add xCAT commands to the path by running the following:

source /etc/profile.d/xcat.sh

Check to see the database is initialized:

tabdump site

The output should similar to the following:

    key,value,comments,disable
    "xcatdport","3001",,
    "xcatiport","3002",,
    "tftpdir","/tftpboot",,
    "installdir","/install",,
         .
         .
         .

If the tabdump command does not work, see Debugging xCAT Problems.

Updating xCAT Packages Later

If you need to update the xCAT RPMs later:

If the management node does not have access to the internet: download the new version of xCAT from Download xCAT and the dependencies from xCAT Dependencies Download and untar them in the same place as before.
If the management node has access to the internet, the commands below will pull the updates directly from the xCAT site.

To update xCAT:

[RH]:

    yum clean metadata
    yum update '*xCAT*'

[SLES]:

    zypper refresh
    zypper update -t package '*xCAT*'

Note: this will not apply updates that may have been made to some of the xCAT deps packages. (If there are brand new deps packages, they will get installed.) In most cases, this is ok, but if you want to make all updates for xCAT rpms and deps, run the following command. This command will also pick up additional OS updates.

[RH]:

    yum update

[SLES]:

    zypper refresh
    zypper update

Note: Sometimes zypper refresh fails to refresh zypper local repository. Try to run zypper clean to clean local metadata, then use zypper refresh.

Note: If you are updating from xCAT 2.7.x (or earlier) to xCAT 2.8 or later, there are some additional migration steps that need to be considered:

Switch from xCAT IBM HPC Integration support to using Software Kits - see
IBM_HPC_Software_Kits#Switching_from_xCAT_IBM_HPC_Integration_Support_to_Using_Software_Kits
for details.
(Optional) Use nic attibutes to replace the otherinterfaces attribute to configure secondary see Cluster_Name_Resolution for details.
Convert non-osimage based system to osimage based system - see
Convert_Non-osimage_Based_System_To_Osimage_Based_System for details

Networks Table

All networks in the cluster must be defined in the networks table. When xCAT was installed, it ran makenetworks, which created an entry in this table for each of the networks the management node is connected to. Now is the time to add to the networks table any other networks in the cluster, or update existing networks in the table.

For a sample Networks Setup, see the following example in Appendix_A
Setting_Up_a_Linux_xCAT_Mgmt_Node#Appendix_A:_Network_Table_Setup_Example

passwd Table

The password should be set in the passwd table that will be assigned to root when the node is installed. You can modify this table using tabedit. To change the default password for root on the nodes, change the system line. To change the password to be used for the BMCs (x-series only), change the ipmi line.

tabedit passwd
#key,username,password,cryptmethod,comments,disable
"system","root","cluster",,,
"hmc","hscroot","ABC123",,,

Setup DNS

To get the hostname/IP pairs copied from /etc/hosts to the DNS on the MN:

Ensure that /etc/sysconfig/named does not have ROOTDIR set
Set site.forwarders to your site-wide DNS servers that can resolve site or public hostnames. The DNS on the MN will forward any requests it can't answer to these servers.

chdef -t site forwarders=1.2.3.4,1.2.5.6
Edit /etc/resolv.conf to point the MN to its own DNS. (Note: this won't be required in xCAT 2.8 and above.)

search cluster
nameserver 172.20.0.1
Run makedns

makedns && service named start

For more information about name resolution in an xCAT Cluster, see [Cluster_Name_Resolution].

Setup TFTP

Nothing to do here - the TFTP server is done by xCAT during the Management Node install.

Setup conserver

makeconservercf

Setup DHCP

You usually don't want your DHCP server listening on your public (site) network, so set site.dhcpinterfaces to your MN's cluster facing NICs. For example:

(Optional)Setup the DHCP interfaces in site table

To set up the site table dhcp interfaces for your system p cluster, identify the correct interfaces that xCAT should listen to on your management node and service nodes:

     chdef -t site dhcpinterfaces='pmanagenode|eth1;service|eth0'
     makedhcp -n
     service dhcpd restart

For Power 775 cluster:

     chdef -t site dhcpinterfaces='pmanagenode|eth1;service|hf0'
     makedhcp -n
     service dhcpd restart

Discover and Define Your System p Hardware

The next steps are to discover your hardware on the nextwork, defined it in the xCAT database, configure xCAT's hardware control, and do the initial definition of the LPARs as the compute nodes. These steps are explained in the following 2 documents. Use the one that applies to your environment:

For Power 775 clusters, use [XCAT_Power_775_Hardware_Management]
For PowerLinux cluster, use [XCAT_PowerLinux_Hardware_Management]
For all other system p hardware, use [XCAT_System_p_Hardware_Management]

After performing the steps in one of those documents, return here and continue on in this document.

Additional Setup on the MN for Power 775 Clusters Only

If you are not setting up a Power 775 cluster, skip this section.

The xCAT Management Node must be configured and running on the DB2 database, before installing the following cluster hardware components.

This section describes the additional setup required for the Power 775 support. This includes the setup of the cluster hardware components and the installation of TEAL, ISNM, and LoadLeveler on the xCAT management node. TEAL, ISNM and LoadLeveler have dependencies on each other so all three must be installed.

Downloading and Installing DFM and hdwr_svr

Refer to the following documentation for downloading and installing DFM and hdwr_svr in an HPC cluster: Downloading and Installing DFM

Obtain additional packages for HFI network

To work with HFI network in Power 775 clusters, the following RPMs and scripts must be obtained from IBM and put on the xCAT MN in the suggested directories. These packages and files should exist as part of the IBM LTC RH6 customized kernel:

/hfi/dd/kernel-2.6.32-*.ppc64.rpm
/hfi/dd/kernel-headers-2.6.32-*.ppc64.rpm
/hfi/dd/hfi_util-*.el6.ppc64.rpm
/hfi/dd/hfi_ndai-*.el6.ppc64.rpm

/hfi/dhcp/net-tools-*.el6.ppc64.rpm
/hfi/dhcp/dhcp-*.el6.ppc64.rpm
/hfi/dhcp/dhclient-*.el6.ppc64.rpm

Install LoadLeveler

Refer to the following documentation for setting up LL in an HPC cluster:

Install Teal

Download the Teal prerequisite rpm packages and the Teal product rpms to your xCAT management. The Teal packages should be available from Teal website: http://sourceforge.net/projects/pyteal/files/ The Teal product does have prerequisites on the LoadL resource manager. Place the packages in a directory such as:

/install/post/otherpkgs/rhels6/ppc64/teal

The Teal prerequisites are:

 gdbm-1.8.3-5
 readline-4.3-2
 python-2.6.2-2
 perl-Module-Load-0.16-115
 pyodbc-2.1.7-1   ***  this is in the latest xCAT deps package

Other than pyodbc, these rpms were most likely installed with your base RedHat installation. If these rpms are not installed, run a command similar to the following to install:

 yum install gdbm.ppc64 readline.ppc64 python.ppc64 perl-Module-Load.ppc64

The Teal rpms for the xCAT EMS are:

teal-base-1.1.0.0-1.ppc64.rpm -used with base Teal support
teal-ll-1.1.0.0-1.ppc64.rpm -used with LL teal
teal-sfp-1.1.0.0-1.ppc64.rpm -used with HMC Service focal point
teal-isnm-1.1.0.0-1.ppc64.rpm -used with isnm Teal
teal-pnsd-1.1.0.0-1.ppc64.rpm -used with PE  pnsd teal

There are other Teal rpms for GPFS that will be required on the Power 775 cluster. GPFS is not required on the EMS, but will be required on GPFS I/O server nodes The teal-gpfs-sn-1.1.0.0-1 rpms has dependencies for gpfs.base and libmmantras.so.

teal-gpfs-1.1.0.0-1.ppc64.rpm     -used with base GPFS
teal-gpfs-sn-1.1.0.0-1.ppc64.rpm  -used with GPFS server nodes

To install:

     yum install pyodbc
     cd /install/post/otherpkgs/<os>/<arch>/teal
     rpm -Uvh ./teal*.rpm

The teal executables are located in /opt/teal/bin directory on the EMS. For more information on Teal please refer to the Teal documentation. (Need to add pointer when available)

Teal tables should viewed using tllsalert and tllsevent commands, or xCAT database commands (e.g. tabedit, tabdump).

For full list of teal commands: https://sourceforge.net/apps/mediawiki/pyteal/index.php?title=Command_Reference

If there are changes to be made for Teal tables, they should be made only using the following teal commands:

tlchalert : close an alert that has been resolved.  It will also close all duplicate alerts that have been reported.
tlrmalert : remove select/all alerts that have been closed that are not associated with other alerts.
tlrmevent : remove select/all events that are not associated with an alert that is still being saved in the alert log.

Typically a user will do these steps to manage Teal alerts and maintain the Teal tables:

Resolve the active/open alerts and then remove them with tlchalert
    tlrmalert --older-than <timestamp> to remove the alerts that are no longer required
    tlrmevent --older-than <timestamp> to remove the events that are no longer required

Install ISNM

Download the ISNM packages to the xCAT MN and place them in a directory such as

    /install/post/otherpkgs/<os>/ppc64/isnm

Install ISNM prerequisite software

Install RSCT

rpm .ivh rsct.core.utils-3.1.0.2-10266.ppc.rpm rsct.core-3.1.0.2-10266.ppc.rpm src-1.3.1.1-10266.ppc.rpm

Obtain RSCT from: https://www14.software.ibm.com/webapp/iwm/web/preLogin.do?lang=en_US&source=stg-rmc

There is one hdwr_svr lib that is not in the package that must be manually placed in /usr/lib on the Managment Node. ( This will be fixed soon). Right now the lib is in the following backing tree:

/project/spreldenali/build/rdenali1107b/export/ppc64_redhat_6.0.0/usr/lib/libnetchmcx.so
cp -p libnetchmcx.so /usr/lib

Discover and define hardware components

The System P hardware components must be discovered, configured and defined in the xCAT database. If you haven't done so already, follow the steps in [XCAT_Power_775_Hardware_Management].

Install and Configure ISNM

Install the ISNM package:

    cd /install/post/otherpkgs/<os>/<arch>/isnm
    rpm -Uvh ./ISNM-cnm*.rpm

Note: you should have already installed the hardware server ISNM-hdwr_svr*.rpm as part of xCAT's DFM.

**NOTE**
There should be pointer to the ISNM documentation in the High Performance Clustering
using 9125-F2C  that describes   HFI-ISR network for Power 775  cluster.
This would be a good place to describe how the HFI is being used with xCAT.
We can provide a pointer where the admin should locate the HFI device drivers.
If you are only trying to communicate over hfi from one octant to another in the same CEC,
the CNM daemon must be running on the EMS.  and the master ISR ID must be loaded to the CEC.
If you are communicating over the HFI from one CEC to another, the HFI cable links
(Dlinks and/or LR links) must be physically configured between the Power 775 CECs.

Check the hardware component and site definitions.

The CNM HFI network requires that the Power 775 frame and cecs be physically installed and properly defined in the xCAT DataBase. This activity should have already been accomplished following the xCAT Power 775 Hardware Management guide. The CNM HFI network requires the following additional data to be defined in the xCAT DB.

Check that the correct Topology has been set in the site table. The topology definition is based on the the number of CECs and type of HFI network configured for your Power 775 cluster.

 lsdef -t site -i topology   (should be one of supported configs 8D, 32D, 128D)
 If there is no topology value found you can set the value with xCAT chdef command
 chdef -t site  topology=32D

Check to make sure that the frame and the CECs node objects have the proper definitions. The frame must be connected to the EMS with DFM, and the frame number is assigned in the "id" attribute. The lsdef frame lists all the frame objects in your cluster, check that each frame has the frame number defined id=<frame #> . If the frame number is not correct, you should execute xCAT command "rspconfig" to set the frame number. The command below will set "id" to 17 for frame17 node, and will update frame number to 17 in the BPA .

 lsdef  frame -i id  (check id attribute to see frame id number assigned for each frame object&gt;
 rspconfig frame17 frame=17
 Note: To change a frame id number with rspconfig, the P775 cecs must be powered off. This activity will take a few minutes
       to complete, and the bpa's will lose HW connections since the BPS'a need to be rebooted to change the configuration.

Check to make sure that the Power 775 cluster has the proper node attributes defined in the xCAT DB.

 lpars/octants - nodes should have hwtype=lpar, hcp and parent has the proper CEC node assigned.
 fsp nodes - should have hwtype=fsp, hcp is set to itself, and parent has proper CEC node assigned.
 cec nodes - should have hwtype=cec, hcp is set to itself, and parent has the proper Frame node assigned.
 bpa nodes - should have hwtype=bpa, hcp is set to itself, and parent has the proper Frame node assigned.
 frame nodes - should have hwtype=frame, hcp is set to itself, and parent is blank or will have a building block number

Check to make sure that the cec node objects have the proper "supernode" attribute defined. The supernode will specify the HFI configuration being used by the cec. You should also make sure the cage id is properly defined where the "id" attribute matches the cage position for the CEC node. The CNM daemon and configuration commands will setup the Master ISR identifier for each cec. This will allow the HFI communications to work between the Power 775 cluster.

 lsdef  cec -i id,supernode  (check that supernode and id attributes are correct for each cec object)
 chdef  f17c01  supernode=0,0   (will set HFI supernode setting)

Hardware server connections

The CNM hardware server daemon will be started as part of the Power 775 Hardware setup working with DFM. The hardware server daemon is used by both xCAT and CNM to track the hardware connections between the xCAT EMS and the Frame/BPA and CEC/FSPs. There are two different connections "tooltype" used with the hardware server daemon and the xCAT mkhwconn command. The tooltype "lpar' is used by the xCAT DFM support, and the tooltype "fnm" is used by the CNM support.

mkhwconn frame17 -t -T fnm   (will make the HW connection for the frame17  frame and cecs (drawers))
mkhwconn cec -t -T fnm

The hardware server daemon works with the /var/opt/isnm/hdwr_svr/data/HmcNetConfig file. The expectation is that HmcNetConfig file will get created as part of the first mkhwconn execution working with xCAT DFM . The CNM will add additional connections for the "fnm" HW connections.

There are hardware server log files that are created and saved under /var/opt/isnm/hdwr_svr/log directory. If you have issues with hardware server daemon, you may want to check the recent "hdwr_svr.log.*" log files. If you need to take a hardware server daemon dump, you can execute "kill -USR1 <hdwr_svr.pid> (this will create the hdwr_svr dump file)

Start CNMD and setup Master ISR ID :

Once all the xCAT definitions are properly updated with CNM configuration data, It is time to start up the CNM daemon and load ounthe proper ISR data in the CECs. Make sure that all the CECs have been powered off prior to the initialization of the CNM daemon. This is necessary to setup the proper HFI configuration data in the Frame and CECs. You can execute the CNM command "chnwm" to activate or deactivate the CNM daemon for AIX EMS

/opt/isnm/cnm/bin/chnwm -d  (take down the CNM daemon)
rpower cec  off             (power down all cecs)
/opt/isnm/cnm/bin/chnwm -a   (activate the CNM daemon)

You can execute the Linux "service" command to activate or deactivite the CNM daemon for Linux EMS.

service cnmd stop     (take down the CNM daemon)
rpower cec off        (power down all cecs)
service cnmd  start   (activate the CNM daemon)

You can now load the HFI master ISR identifier on the Power 775 frame and cecs. This will be accomplished using the ISNM command "chnwsvrconfig". You will need to specify this command for your frame/cecs in you P775 cluster.

/opt/isnm/cnm/bin/chnwsvrconfig  -A   (will configure all associated P775 Frames/cecs with HFI data)
/opt/isnm/cnm/bin/chnwsvrconfig  -f 17 -c 3   (will configure one cec in P775 frame 17 with cage id 3)

You can now verify that the CNM daemon and HFI configuration is working by executing the CNM command "nmcmd" to dump the drawer status information. This will list the current state of the drawers working with CNM. Please reference the HPC using the 9125-F2C guide for more detail about CNM commands, implementation, and debug.

 /opt/isnm/cnm/bin/nmcmd -D -D
 # nmcmd -D -D
 Frame 17  Cage 3 Supernode 0 Drawer 0 STANDBY
 Frame 17  Cage 5 Supernode 0 Drawer 2 STANDBY
 Frame 17  Cage 4 Supernode 0 Drawer 1 STANDBY

Configure DFM Hierarchically (Optional)

Depending on how large your cluster is, you may find that DFM performs better if the operations are sent to the FSPs via the service nodes. See Overview of Using DFM Hierarchically to get an overview of how to use DFM Hierarchically.

Setup the xCAT MN for a Hierarchical Cluster

For large clusters, you can distribute most of the xCAT services from the management node to xCAT service nodes. Doing so, creates a "hierarchical cluster". The following document describes additional xCAT management node configuration for a Linux Hierarchical Cluster, how to install your service nodes, and how to configure your compute nodes to be managed by service nodes.

[Setting_Up_a_Linux_Hierarchical_Cluster]

Note: Hierarchical Clusters are required for Power 775 clusters in order to deploy and manage compute nodes across the HFI network. In addition to setting up xCAT cluster hierarchy, the above document also contains important instructions specific to configuring and managing your Power 775 cluster.

Setting up a hierarchical cluster assumes more advanced knowledge of xCAT node definition, deployment, and management. If this is your first time setting up an xCAT cluster, you should skip this step for now and experiment with a simple cluster managed directly from your xCAT management node to become familiar with all the different concepts and processes.

Complete the Definition of the Compute Nodes

The hardware management documents explained how to get the LPARs of the CECs defined as nodes in the xCAT database. Before deploying an OS on the nodes, you must set some additional attributes of the nodes.

Define xCAT groups

See the following [Node_Group_Support],for more details on how to define xCAT groups. For the example below add the compute group to the nodes.

chdef -t node -o pnode1,pnode2 -p groups=compute

Update the attributes of the node

chdef -t node -o pnode1 netboot=yaboot tftpserver=192.168.0.1 nfsserver=192.168.0.1
monserver=192.168.0.1 xcatmaster=192.168.0.1 installnic="eth0" primarynic="eth0"

Note: Make sure the attributes "installnic" and "primarynic" are set up by the correct Ethernet or HFI Interface of compute node. Otherwise the compute node installation may hang on requesting information from an incorrect interface. The "installnic" and "primarynic" can also be set to mac address if you are not sure about the Ethernet interface name, the mac address can be got through getmacs command. The installnic" and "primarynic" can also be set to keyword "mac", which means that the network interface specified by the mac address in the mac table will be used.

Make sure that the address used above ( 192.168.0.1) is the address of the Management Node as known by the node. Also make sure site.master has this address.

If you are using redhat 7 (RHEL7), "yaboot" is deprecated, the netboot method for system P is must be "grub2"Make sure that the address used above ( 192.168.0.1) is the address of the Management Node as known by the node. Also make sure site.master has this address.
. So, set "netboot" attribute to "grub2" to provision Redhat 7 on ppc64 node.

Check the site.master value

Make sure site.master is the address or name known by the node

To change site.master to this address:

chdef -t site -o clustersite master="192.168.0.1"

Set the type attributes of the node

    chdef -t node -o pnode1 os=<os> arch=ppc64 profile=compute

For valid options:

 tabdump -d nodetype

Configure conserver

The xCAT rcons command uses the conserver package to provide support for multiple read-only consoles on a single node and the console logging. For example, if a user has a read-write console session open on node node1, other users could also log in to that console session on node1 as read-only users. This allows sharing a console server session between multiple users for diagnostic or other collaborative purposes. The console logging function will log the console output and activities for any node with remote console attributes set to the following file which an be replayed for debugging or any other purposes:

    /var/log/consoles/<management node>

Note: conserver=<management node> is the default, so it optional in the command

Update conserver configuration

Each xCAT node with remote console attributes set should be added into the conserver configuration file to make the rcons work. The xCAT command makeconservercf will put all the nodes into conserver configuration file /etc/conserver.cf. The makeconservercf command must be run when there is any node definition changes that will affect the conserver, such as adding new nodes, removing nodes or changing the nodes' remote console settings.

To add or remove new nodes for conserver support:

makeconservercf
service conserver stop
service conserver start

Check rcons(rnetboot and getmacs depend on it)

The functions rnetboot and getmacs depend on conserver functions, check it is available.

rcons pnode1

If it works ok, you will get into the console interface of the pnode1. If it does not work, review your rcons setup as documented in previous steps.

Check hardware control setup to the nodes

See if you setup is correct at this point, run rpower to check node status:

rpower pnode1 stat

Update the mac table with the address of the node(s)

Before run getmacs, make sure the node is off. The reason is the HMC cannot shutdown linux nodes which are in running state.

You can force the lpar shutdown with:

rpower pnode1 stat, if node is on then run
rpower pnode1 off

If there's only one Ethernet adapter on the node or you have specified the installnic or primarynic attribute of the node, using following command can get the correct mac address.

Check for *nic definition, by running

lsdef pnode1

To set installnic or primarynic:

chdef -t pnode1 -o blade01 installnic=eth0 primarynic=eth1

Get mac addresses:

getmacs pnode1

If there are more than one Ethernet adapters on the node, and you don't know which one has been configured for the installation process, or the lpar is just created and there is no active profile for that lpar, or the lpar is on a P5 system and there is no lhea/sea ethernet adapters, then you have to specify more parameters for the lpar to try to figure out an available interface by using the ping operation. Run this command:

getmacs pnode1 -D -S 192.168.0.1 -G 192.168.0.10

The output looks like following:

pnode1:
Type Location Code MAC Address Full Path Name Ping Result Device Type
ent U9133.55A.10E093F-V4-C5-T1 f2:60:f0:00:40:05 /vdevice/l-lan@30000005 virtual

And the Mac address will be written into the xCAT mac table. Run to verify:

tabdump mac

Update the mac table with the address of the node(s) for Power 775

To set installnic or primarynic:

chdef -t node -o c250f07c04ap13 installnic=hf0 primarynic=hf1

Get mac addresses:

getmacs c250f07c04ap13 -D

Configure DHCP

Add the defined nodes into the DHCP configuration:

 makedhcp c250f07c04ap13

Restart the dhcp service:

 service dhcpd restart

Set up customization scripts (optional)

xCAT supports the running of customization scripts on the nodes when they are installed. You can see what scripts xCAT will run by default by looking at the xcatdefaults entry in the xCAT postscripts database table. The postscripts attribute of the node definition can be used to specify the comma separated list of the scripts that you want to be executed on the nodes. The order of the scripts in the list determines the order in which they will be run.

To check current postscript and postbootscripts setting:

tabdump postscripts

For example, if you want to have your two scripts called foo and bar run on node node01 you could add them to the postscripts table:

**chdef -t node -o node01 -p postscripts=foo,bar**

(The -p flag means to add these to whatever is already set.)

For more information on creating and setting up Post*scripts: [Postscripts_and_Prescripts]

Install Stateful Nodes

This section describes deploying stateful nodes.
There are two options to install your nodes as stateful (diskful) nodes:

Use ISOs or DVDs, follow the Option 1 , Installing Stateful Nodes using ISOs or DVDs below.
Clone new nodes from a pre-installed/pre-configured node, follow the Option 2,Installing Stateful Nodes Using Sysclone below.

Option 1: Installing Stateful Nodes Using ISOs or DVDs
Option 2: Installing Stateful Nodes Using Sysclone
- Install or Configure the Golden Client
- Capture image from the Golden Client

Option 1: Installing Stateful Nodes Using ISOs or DVDs

This section describes the process for setting up xCAT to install nodes; that is how to install an OS on the disk of each node.

Create the Distro Repository on the MN

The copycds command copies the contents of the linux distro media to /install/<os>/<arch> so that it will be available to install nodes with or create diskless images.

Obtain the Redhat or SLES ISOs or DVDs.
If using an ISO, copy it to (or NFS mount it on) the management node, and then run:
```
copycds <path>/RHEL6.2-*-Server-x86_64-DVD1.iso
```

If using a DVD, put it in the DVD drive of the management node and run:

copycds /dev/dvd       # or whatever the device name of your dvd drive is

Tip: if this is the same distro version as your management node, create a .repo file in /etc/yum.repos.d with content similar to:

    [local-rhels6.2-x86_64]
    name=xCAT local rhels 6.2
    baseurl=file:/install/rhels6.2/x86_64
    enabled=1
    gpgcheck=0

This way, if you need some additional RPMs on your MN at a later, you can simply install them using yum. Or if you are installing other software on your MN that requires some additional RPMs from the disto, they will automatically be found and installed.

Select or Create an osimage Definition

The copycds command also automatically creates several osimage defintions in the database that can be used for node deployment. To see them:

    lsdef -t osimage          # see the list of osimages
    lsdef -t osimage <osimage-name>          # see the attributes of a particular osimage

From the list above, select the osimage for your distro, architecture, provisioning method (in this case install), and profile (compute, service, etc.). Although it is optional, we recommend you make a copy of the osimage, changing its name to a simpler name. For example:

    lsdef -t osimage -z rhels6.2-x86_64-install-compute | sed 's/^[^ ]\+:/mycomputeimage:/' | mkdef -z

This displays the osimage "rhels6.2-x86_64-install-compute" in a format that can be used as input to mkdef, but on the way there it uses sed to modify the name of the object to "mycomputeimage".

Initially, this osimage object points to templates, pkglists, etc. that are shipped by default with xCAT. And some attributes, for example otherpkglist and synclists, won't have any value at all because xCAT doesn't ship a default file for that. You can now change/fill in any osimage attributes that you want. A general convention is that if you are modifying one of the default files that an osimage attribute points to, copy it into /install/custom and have your osimage point to it there. (If you modify the copy under /opt/xcat directly, it will be over-written the next time you upgrade xCAT.)

But for now, we will use the default values in the osimage definition and continue on. (If you really want to see examples of modifying/creating the pkglist, template, otherpkgs pkglist, and sync file list, see the section [Using_Provmethod=osimagename]. Most of the examples there can be used for stateful nodes too.)

Install a New Kernel on the Nodes (Optional)

Create a postscript file called (for example) updatekernel:

    vi /install/postscripts/updatekernel

Add the following lines to the file:

    #!/bin/bash
    rpm -Uivh data/kernel-*rpm

Change the permission on the file:

    chmod 755 /install/postscripts/updatekernel

Make the new kernel RPM available to the postscript:

    mkdir /install/postscripts/data
    cp <kernel> /install/postscripts/data

Add the postscript to your compute nodes:

    chdef -p -t group compute postscripts=updatekernel

Now when you install your nodes (done in a step below), it will also update the kernel.

Alternatively, you could install your nodes with the stock kernel, and update the nodes afterward using updatenode and the same postscript above, in this case, you need to reboot your nodes to make the new kernel be effective.

Customize the disk partitioning (Optional)

By default, xCAT will install the operating system on the first disk in the node. However, you may choose to customize the disk partitioning during the install process and define a specific disk layout. You can do this in one of two ways:

1. Partitioning definition configuration file: Create a custom osimage file that contains the correct disk partitioning definition. The nodeset command will insert the contents of this file directly into the generated autoinst configuration file that will be used by the OS installer (e.g. kickstart for RedHat, AutoYaST for SLES). The file must be formatted for that installer. For example, for a SLES AutoYaST install, you might create a file called /install/custom/my-partitions, containing:

  <drive>
     <device>XCATPARTITIONHOOK</device>
     <initialize config:type="boolean">true</initialize>
     <use>all</use>
     <partitions config:type="list">
       <partition>
         <create config:type="boolean">true</create>
         <filesystem config:type="symbol"<swap>/filesystem>
         <format config:type="boolean">true</format>
         <mount>swap</mount>
         <mountby config:type="symbol"<path>/mountby>
         <partition_nr config:type="integer"<1>/partition_nr>
         <partition_type>primary</partition_type>
         <size>32G</size>
       </partition>
       <partition>
         <create config:type="boolean">true</create>
         <filesystem config:type="symbol">ext3</filesystem>
         <format config:type="boolean">true</format>
         <mount>/</mount>
         <mountby config:type="symbol">path</mountby>
         <partition_nr config:type="integer"<2>/partition_nr>
         <partition_type>primary</partition_type>
         <size>64G</size>
       </partition>
     </partitions>
   </drive>

Then use this file in your osimage:

      chdef -t osimage <osimagename> partitionfile=/install/custom/my-partitions

When nodeset runs and generates the /install/autoinst file for a node, it will replace the #XCAT_PARTITION_START#...#XCAT_PARTITION_END# directives from your osimage template with the contents of your custom partitionfile.

2. Partitioning definition script: Create a shell script that will be run on the node during the install process to dynamically create the disk partitioning definition. This script will be run during the OS installer %pre script execution and must write the correct partitioning definition into the file /tmp/partitionfile on the node. For example, for a RedHat kickstart install, you might create a file called /install/custom/my-partitions.sh, containing:

     echo "part swap --size 1024" > /tmp/partitionfile
     echo "part / --size 1 --grow --fstype ext3" >> /tmp/partitionfile

Then assign this script to your osimage:

     chdef -t osimage <osimagename> partitionfile='s:/install/custom/my-partitions.sh'

Note the 's:' preceding the filename tells nodeset that this is a script. When nodeset runs and generates the /install/autoinst file for a node, it will add the execution of the contents of this script to the %pre section of that file. The nodeset command will then replace the #XCAT_PARTITION_START#...#XCAT_PARTITION_END# directives from the osimage template file with "%include /tmp/partitionfile" to dynamically include the tmp definition file your script created.

Update the Distro at a Later Time

After the initial install of the distro onto nodes, if you want to update the distro on the nodes (either with a few updates or a new SP) without reinstalling the nodes:

create the new repo using copycds:
```
copycds <path>/RHEL6.3-*-Server-x86_64-DVD1.iso
```
Or, for just a few updated rpms, you can copy the updated rpms from the distributor into a directory under /install and run createrepo in that directory.
add the new repo to the pkgdir attribute of the osimage:
```
chdef -t osimage rhels6.2-x86_64-install-compute -p pkgdir=/install/rhels6.3/x86_64
```
Note: the above command will add a 2nd repo to the pkgdir attribute. This is only supported for xCAT 2.8.2 and above. For earlier versions of xCAT, omit the -p flag to replace the existing repo directory with the new one.
run the ospkgs postscript to have yum update all rpms on the nodes
```
updatenode compute -P ospkgs
```

Option 2: Installing Stateful Nodes Using Sysclone

This section describes how to install or configure a diskful node (we call it a golden-client), capture an osimage from this golden-client, then the osimage can be used to install/clone other nodes. See Using_Clone_to_Deploy_Server for more information.

Note: this support is available in xCAT 2.8.2 and above.

Install or Configure the Golden Client

If you want to use the sysclone provisioning method, you need a golden-client. In this way, you can customize and tweak the golden-client’s software and configuration according to your needs, and verify it’s proper operation. Once the image is captured and deployed, the new nodes will behave in the same way the golden-client does.

To install a golden-client, follow the section Installing_Stateful_Linux_Nodes#Option_1:_Installing_Stateful_Nodes_Using_ISOs_or_DVDs.

To install the systemimager rpms on the golden-client, do these steps on the mgmt node:

Download the xcat-dep tarball which includes systemimager rpms. (You might already have the xcat-dep tarball on the mgmt node.)

Go to xcat-dep and get the latest xCAT dependency tarball. Copy the file to the management node and untar it in the appropriate sub-directory of /install/post/otherpkgs. For example:

(For RH/CentOS):    
mkdir -p /install/post/otherpkgs/rhels6.3/x86_64/xcat
cd /install/post/otherpkgs/rhels6.3/x86_64/xcat
tar jxvf xcat-dep-*.tar.bz2

(For SLES):  
mkdir -p /install/post/otherpkgs/sles11.3/x86_64/xcat
cd /install/post/otherpkgs/sles11.3/x86_64/xcat
tar jxvf xcat-dep-*.tar.bz2

Add the sysclone otherpkglist file and otherpkgdir to osimage definition that is used for the golden client, and then use updatenode to install the rpms. For example:

(For RH/CentOS): 
chdef -t osimage -o <osimage-name> otherpkglist=/opt/xcat/share/xcat/install/rh/sysclone.rhels6.x86_64.otherpkgs.pkglist
chdef -t osimage -o <osimage-name> -p otherpkgdir=/install/post/otherpkgs/rhels6.3/x86_64
updatenode <my-golden-cilent> -S

(For SLES):  
chdef -t osimage -o <osimage-name> otherpkglist=/opt/xcat/share/xcat/install/sles/sysclone.sles11.x86_64.otherpkgs.pkglist
chdef -t osimage -o <osimage-name> -p otherpkgdir=/install/post/otherpkgs/sles11.3/x86_64
updatenode <my-golden-cilent> -S

Capture image from the Golden Client

On the mgmt node, use imgcapture to capture an osimage from the golden-client.

    imgcapture <my-golden-client> -t sysclone -o <mycomputeimage>

Tip: when imgcapture is run, it pulls the osimage from the golden-client, and creates the image files system and a corresponding osimage definition on the xcat management node.

 lsdef -t osimage <mycomputeimage> to check the osimage attributes.

Begin Installation

Use network boot to start the installation

rnetboot compute

Alternative network boot in Power 775

For Power 775 nodes, you can also initiate network boot using the rbootseq and rpower commands, but this method is not recommended for diskfull p775 service nodes.

Monitor installation

It is possible to use the wcons command to watch the installation process for a sampling of the nodes:

wcons n1,n20,n80,n100

or rcons to watch one node

rcons n1

Additionally, nodestat may be used to check the status of a node as it installs:

nodestat n20,n21
n20: installing man-pages - 2.39-10.el5 (0%)
n21: installing prep

Note: the percentage complete reported by nodestat is not necessarily reliable.

You can also watch nodelist.status until it changes to "booted" for each node:

nodels compute nodelist.status | xcoll

Once all of the nodes are installed and booted, you should be able ssh to all of them from the MN (w/o a password), because xCAT should have automatically set up the ssh keys (if the postscripts ran successfully):

xdsh compute date

If there are problems, see [Debugging_xCAT_Problems].

Installing Additional Packages Using an Otherpkgs Pkglist

If you have additional rpms (rpms not in the distro) that you also want installed, make a directory to hold them, create a list of the rpms you want installed, and add that information to the osimage definition:

Create a directory to hold the additional rpms:

mkdir -p /install/post/otherpkgs/rh/x86_64
cd /install/post/otherpkgs/rh/x86_64
cp /myrpms/* .
createrepo .

NOTE: when the management node is rhels6.x, and the otherpkgs repository data is for rhels5.x, we should run createrepo with "-s md5". Such as:

createrepo -s md5 .

Create a file that lists the additional rpms that should be installed. For example, in /install/custom/netboot/rh/compute.otherpkgs.pkglist put:

myrpm1
myrpm2
myrpm3
Add both the directory and the file to the osimage definition:

chdef -t osimage mycomputeimage otherpkgdir=/install/post/otherpkgs/rh/x86_64 otherpkglist=/install/custom/netboot/rh/compute.otherpkgs.pkglist

If you add more rpms at a later time, you must run createrepo again. The createrepo command is in the createrepo rpm, which for RHEL is in the 1st DVD, but for SLES is in the SDK DVD.

If you have multiple sets of rpms that you want to keep separate to keep them organized, you can put them in separate sub-directories in the otherpkgdir. If you do this, you need to do the following extra things, in addition to the steps above:

Run createrepo in each sub-directory
In your otherpkgs.pkglist, list at least 1 file from each sub-directory. (During installation, xCAT will define a yum or zypper repository for each directory you reference in your otherpkgs.pkglist.) For example:

xcat/xcat-core/xCATsn
xcat/xcat-dep/rh6/x86_64/conserver-xcat

There are some examples of otherpkgs.pkglist in /opt/xcat/share/xcat/netboot/<distro>/service.*.otherpkgs.pkglist that show the format.

Note: the otherpkgs postbootscript should by default be associated with every node. Use lsdef to check:

lsdef node1 -i postbootscripts

If it is not, you need to add it. For example, add it for all of the nodes in the "compute" group:

chdef -p -t group compute postbootscripts=otherpkgs

Installing OS Updates By Setting linuximage.pkgdir(only support for rhels and sles)

The linuximage.pkgdir is the name of the directory where the distro packages are stored. It can be set multiple paths. The multiple paths must be separated by ",". The first path is the value of osimage.pkgdir and must be the OS base pkg directory path, such as pkgdir=/install/rhels6.2/x86_64,/install/updates/rhels6.2/x86_64 . In the os base pkg path, there is default repository data. In the other pkg path(s), the users should make sure there is repository data. If not, use "createrepo" command to create them.

If you have additional os updates rpms (rpms may be from the os website, or the additional os distro) that you also want installed, make a directory to hold them, create a list of the rpms you want installed, and add that information to the osimage definition:

Create a directory to hold the additional rpms:

mkdir -p /install/updates/rhels6.2/x86_64 
cd /install/updates/rhels6.2/x86_64 
cp /myrpms/* .

If there is no repository data in the directory, you can run "createrepo" to create it:

    createrepo .

The createrepo command is in the createrepo rpm, which for RHEL is in the 1st DVD, but for SLES is in the SDK DVD.

NOTE: when the management node is rhels6.x, and the otherpkgs repository data is for rhels5.x, we should run createrepo with "-s md5". Such as:

    createrepo -s md5 .

Append the additional rpms into the corresponding pkglist. For example, in /install/custom/install/rh/compute.rhels6.x86_64.pkglist, append:
```
...
myrpm1
myrpm2
myrpm3
```

Add both the directory and the file to the osimage definition:

chdef -t osimage mycomputeimage pkgdir=/install/rhels6.2/x86_64,/install/updates/rhels6.2/x86_64 \
    pkglist=/install/custom/install/rh/compute.rhels6.x86_64.pkglist

If you add more rpms at a later time, you must run createrepo again.

Note: After the above setting,

For diskfull install, run "nodeset <noderange> mycomputeimage" to pick up the changes, and then boot up the nodes
For diskless, run genimage to install the packages into the image, and then packimage and boot up the nodes.
If the nodes are up, run "updatenode <noderange> ospkgs" to update the packages.
These functions are only support for rhels6.x and sles11.x

Stateless Node Deployment

The following section (and its subsections) is the standard xCAT procedure for building and deploying a linux stateless image. Some of the example commands refer to the x86_64 architecture, but the procedure is the same on ppc64. Just replace x86_64 with ppc64. Also, when it comes time to boot the nodes, use rnetboot instead of rpower.

In addition, if you are building/deploying a stateless image on a p775 cluster, do these additional things when following the procedure below:

In the section for installing other packages, add the powerpc-utils rpm to the otherpkgs directory and the otherpkglist.
In the section for using postinstall files, add the following lines to your postinstall script (the location of the rootimg should be changed to your location):

cp /hfi/dd/ /install/test/netboot/rh/ppc64/compute/rootimg/tmp/
chroot /install/test/netboot/rh/ppc64/compute/rootimg/ /bin/rpm -ivh '/tmp/dhclient-.rpm' --force
chroot /install/test/netboot/rh/ppc64/compute/rootimg/ /bin/rpm -ivh '/tmp/dhcp-.rpm' --force
chroot /install/test/netboot/rh/ppc64/compute/rootimg/ /bin/rpm -ivh '/tmp/kernel-headers-.rpm' --force
chroot /install/test/netboot/rh/ppc64/compute/rootimg/ /bin/rpm -ivh '/tmp/net-tools-.rpm' --force
chroot /install/test/netboot/rh/ppc64/compute/rootimg/ /bin/rpm -ivh '/tmp/hfi_ndai-.rpm' --force
chroot /install/test/netboot/rh/ppc64/compute/rootimg/ /bin/rpm -ivh '/tmp/hfi_util-*.rpm' --force
In the Generate/Pack image section, put the p775 custom kernel in /install/kernels and use a genimage command like:

genimage -i hf0 -n hf_if -k 2.6.32-71.el6.20110617.ppc64 redhat6img
Also verify that name resolution will be set up correctly on the nodes. This is necessary for the confighfi postscript to configure the HFI NICs properly. See [Cluster_Name_Resolution] for details about setting up name resolution.
When it it comes time to boot the nodes, use rbootseq and rpower.

Note: this section describes how to create a stateless image using the genimage command to install a list of rpms into the image. As an alternative, you can also capture an image from a running node and create a stateless image out of it. See [Capture_Linux_Image] for details.

Create the Distro Repository on the MN
Using an osimage Definition
Select or Create an osimage Definition
Set up pkglists
Set up a postinstall script (optional)
Set up Files to be synchronized on the nodes
Configure the nodes to use your osimage
Generate and pack your image
Boot the nodes

Create the Distro Repository on the MN

The copycds command copies the contents of the linux distro media to /install/<os>/<arch> so that it will be available to install nodes with or create diskless images.

Obtain the Redhat or SLES ISOs or DVDs.
If using an ISO, copy it to (or NFS mount it on) the management node, and then run:

copycds <path>/RHEL6.2-Server-20080430.0-x86_64-DVD.iso
If using a DVD, put it in the DVD drive of the management node and run:

copycds /dev/dvd # or whatever the device name of your dvd drive is

Tip: if this is the same distro version as your management node, create a .repo file in /etc/yum.repos.d with content similar to:

[local-rhels6.2-x86_64]
name=xCAT local rhels 6.2
baseurl=file:/install/rhels6.2/x86_64
enabled=1
gpgcheck=0

Using an osimage Definition

Note: To use an osimage as your provisioning method, you need to be running xCAT 2.6.6 or later.

The provmethod attribute of your nodes should contain the name of the osimage object definition that is being used for those nodes. The osimage object contains paths for pkgs, templates, kernels, etc. If you haven't already, run copycds to copy the distro rpms to /install. Default osimage objects are also defined when copycds is run. To view the osimages:

    lsdef -t osimage          # see the list of osimages
    lsdef -t osimage <osimage-name>
          # see the attributes of a particular osimage

Select or Create an osimage Definition

From the list found above, select the osimage for your distro, architecture, provisioning method (install, netboot, statelite), and profile (compute, service, etc.). Although it is optional, we recommend you make a copy of the osimage, changing its name to a simpler name. For example:

    lsdef -t osimage -z rhels6.3-x86_64-netboot-compute | sed 's/^[^ ]\+:/mycomputeimage:/' | mkdef -z

This displays the osimage "rhels6.3-x86_64-netboot-compute" in a format that can be used as input to mkdef, but on the way there it uses sed to modify the name of the object to "mycomputeimage".

      chdef -t osimage -o mycomputeimage rootimgdir=/install/netboot/rhels6.3/x86_64/mycomputeimage

Set up pkglists

You likely want to customize the main pkglist for the image. This is the list of rpms or groups that will be installed from the distro. (Other rpms that they depend on will be installed automatically.) For example:

    mkdir -p /install/custom/netboot/rh
    cp -p /opt/xcat/share/xcat/netboot/rh/compute.rhels6.x86_64.pkglist /install/custom/netboot/rh
    vi /install/custom/netboot/rh/compute.rhels6.x86_64.pkglist
    chdef -t osimage mycomputeimage pkglist=/install/custom/netboot/rh/compute.rhels6.x86_64.pkglist

The goal is to install the fewest number of rpms that still provides the function and applications that you need, because the resulting ramdisk will use real memory in your nodes.

Also, check to see if the default exclude list excludes all files and directories you do not want in the image. The exclude list enables you to trim the image after the rpms are installed into the image, so that you can make the image as small as possible.

    cp /opt/xcat/share/xcat/netboot/rh/compute.exlist /install/custom/netboot/rh
    vi /install/custom/netboot/rh/compute.exlist 
    chdef -t osimage mycomputeimage exlist=/install/custom/netboot/rh/compute.exlist

Make sure nothing is excluded in the exclude list that you need on the node. For example, if you require perl on your nodes, remove the line "./usr/lib/perl5*".

[[include Install_OS_Updates (already included)]

[[include Install_Additional_Packages (already included)]

Set up a postinstall script (optional)

Postinstall scripts for diskless images are analogous to postscripts for diskfull installation. The postinstall script is run by genimage near the end of its processing. You can use it to do anything to your image that you want done every time you generate this kind of image. In the script you can install rpms that need special flags, or tweak the image in some way. There are some examples shipped in /opt/xcat/share/xcat/netboot/<distro>. If you create a postinstall script to be used by genimage, then point to it in your osimage definition. For example:

    chdef -t osimage mycomputeimage postinstall=/install/custom/netboot/rh/compute.postinstall

Set up Files to be synchronized on the nodes

Note: This is only supported for stateless nodes in xCAT 2.7 and above.

Sync lists contain a list of files that should be sync'd from the management node to the image and to the running nodes. This allows you to have 1 copy of config files for a particular type of node and make sure that all those nodes are running with those config files. The sync list should contain a line for each file you want sync'd, specifying the path it has on the MN and the path it should be given on the node. For example:

    /install/custom/syncfiles/compute/etc/motd > /etc/motd
    /etc/hosts > /etc/hosts

If you put the above contents in /install/custom/netboot/rh/compute.synclist, then:

    chdef -t osimage mycomputeimage synclists=/install/custom/netboot/rh/compute.synclist

For more details, see Sync-ing_Config_Files_to_Nodes.

Configure the nodes to use your osimage

You can configure any noderange to use this osimage. In this example, we define that the whole compute group should use the image:

     chdef -t group compute provmethod=mycomputeimage

Now that you have associated an osimage with nodes, if you want to list a node's attributes, including the osimage attributes all in one command:

    lsdef node1 --osimage

Generate and pack your image

There are other attributes that can be set in your osimage definition. See the osimage man page for details.

Building an Image for a Different OS or Architecture

If you are building an image for a different OS/architecture than is on the Management node, you need to follow this process: [Building_a_Stateless_Image_of_a_Different_Architecture_or_OS]. Note: different OS in this case means, for example, RHEL 5 vs. RHEL 6. If the difference is just an update level/service pack (e.g. RHEL 6.0 vs. RHEL 6.3), then you can build it on the MN.

Building an Image for the Same OS and Architecture as the MN

If the image you are building is for nodes that are the same OS and architecture as the management node (the most common case), then you can follow the instructions here to run genimage on the management node.

Run genimage to generate the image based on the mycomputeimage definition:

    genimage mycomputeimage

Before you pack the image, you have the opportunity to change any files in the image that you want to, by cd'ing to the rootimgdir (e.g. /install/netboot/rhels6/x86_64/compute/rootimg). Although, instead, we recommend that you make all changes to the image via your postinstall script, so that it is repeatable.

The genimage command creates /etc/fstab in the image. If you want to, for example, limit the amount of space that can be used in /tmp and /var/tmp, you can add lines like the following to it (either by editing it by hand or via the postinstall script):

    tmpfs   /tmp     tmpfs    defaults,size=50m             0 2
    tmpfs   /var/tmp     tmpfs    defaults,size=50m       0 2

But probably an easier way to accomplish this is to create a postscript to be run when the node boots up with the following lines:

    logger -t xcat "$0: BEGIN"
    mount -o remount,size=50m /tmp/
    mount -o remount,size=50m /var/tmp/
    logger -t xcat "$0: END"

Assuming you call this postscript settmpsize, you can add this to the list of postscripts that should be run for your compute nodes by:

    chdef -t group compute -p postbootscripts=settmpsize

Now pack the image to create the ramdisk:

    packimage mycomputeimage

Installing a New Kernel in the Stateless Image

The kerneldir attribute in linuximage table is used to assign one directory to hold the new kernel to be installed into the stateless/statelite image. Its default value is /install/kernels, you need to create the directory named <kernelver> under the kerneldir, and genimage will pick them up from there.

Assuming you have the kernel in RPM format in /tmp, the value of kerneldir is not set (which will take the default value: /install/kernels).

This procedure assumes you are using xCAT 2.6.1 or later. The rpm names are an example and you can substitute your level and architecture. The kernel will be installed directly from the rpm package.

For RHEL:

The kernel RPM package is usually named kernel-<kernelver>.rpm, for example: kernel-2.6.32.10-0.5.x86_64.rpm is the kernel package for 2.6.32.10-0.5.x86_64.

    cp /tmp/kernel-2.6.32.10-0.5.x86_64.rpm /install/kernels/
    createrepo /install/kernels/

For SLES:

Usually, the kernel files for SLES are separated into two parts: kernel-<arch>-base and kernel, and the naming of kernel RPM packages are different. For example, there's two RPM packages in /tmp:

    kernel-ppc64-base-2.6.27.19-5.1.x86_64.rpm
    kernel-ppc64-2.6.27.19-5.1.x86_64.rpm

2.6.27.19-5.1.x86_64 is NOT the kernel version. _2.6.27.19-5-_x86_64 is the kernel version . Follow this naming convention to determine the kernel version.

After the kernel version is determined for SLES, then:

    cp /tmp/kernel-ppc64-base-2.6.27.19-5.1.x86_64.rpm /install/kernels/
    cp /tmp/kernel-ppc64-2.6.27.19-5.1.x86_64.rpm /install/kernels/

Run genimage/packimage to update the image with the new kernel: (Use sles as example)

Since the kernel version is different from the rpm package version, the -g flag needs to be specified on the genimage command for the rpm version of kernel packages.

    genimage -i eth0 -n ibmveth -o sles11.1 -p compute -k 2.6.27.19-5-x86_64 -g 2.6.27.19-5.1

Installing New Kernel Drivers to Stateless Initrd

The kernel drivers in the stateless initrd are used for the devices during the netboot. If you are missing one or more kernel drivers for specific devices (especially for the network device), the netboot process will fail. xCAT offers two approaches to add additional drivers to the stateless initrd during the running of genimage.

Use the '-n' flag to add new drivers to the stateless initrd
```
genimage <imagename> -n <new driver list>
```

Generally, the genimage command has a default driver list which will be added to the initrd. But if you specify the '-n' flag, the default driver list will be replaced with your <new driver list>. That means you need to include any drivers that you need from the default driver list into your <new driver list>.

The default driver list:

    rh-x86:   tg3 bnx2 bnx2x e1000 e1000e igb mlx_en virtio_net be2net
    rh-ppc:   e1000 e1000e igb ibmveth ehea
    sles-x86: tg3 bnx2 bnx2x e1000 e1000e igb mlx_en be2net
    sels-ppc: tg3 e1000 e1000e igb ibmveth ehea be2net

Note: With this approach, xCAT will search for the drivers in the rootimage. You need to make sure the drivers have been included in the rootimage before generating the initrd. You can install the drivers manually in an existing rootimage (using chroot) and run genimage again, or you can use a postinstall script to install drivers to the rootimage during your initial genimage run.

Use the driver rpm package to add new drivers from rpm packages to the stateless initrd

Refer to the doc Using_Linux_Driver_Update_Disk#Driver_RPM_Package.

Boot the nodes

    nodeset compute osimage=mycomputeimage

(If you need to update your diskless image sometime later, change your osimage attributes and the files they point to accordingly, and then rerun genimage, packimage, nodeset, and boot the nodes.)

Now boot your nodes...

Use network boot to start the installation for non-p775 nodes

rnetboot compute

Use network boot to start the installation for p775 nodes

Starting with xCAT 2.6, Power 775 diskless nodes can also be network booted using the rbootseq and rpower commands. This is recommended (instead of rnetboot) for diskless p775 nodes, because it is faster:

rbootseq compute hfi
rpower compute on

Note: if your diskless p775 node has an ethernet adapter and you are trying to network boot it via that NIC, instead of the HFI NIC, then use "net" instead of "hfi" as the argument to the rbootseq command. This is an unusual case.

Check the installation result

After the node installation is completed successfully, the node's status will be changed to booted, the following command to check the node's status:

lsdef compute -i status

When the node's status is changed to booted, you can also check ssh service on the node is working and you can login without password.

Note: Do not run ssh or xdsh against the node until the node installation is completed successfully. Running ssh or xdsh against the node before the node installation completed may result in ssh hostkeys issues.

If ssh is working but cannot login without password, setup the ssh key to the compute node using xdsh:

xdsh compute -K

After exchanging ssh key, following command should work.

xdsh compute date

Remove an image

If you want to remove an image, rmimage is used to remove the Linux stateless or statelite image from the file system. It is better to use this command than just remove the filesystem yourself, because it also remove appropriate links to real files system that may be distroyed on your Management Node, if you just use the rm -rf command.

You can specify the <os>, <arch> and <profile> value to the rmimagecommand:

    rmimage -o <os> -a <arch> -p <profile>

Or, you can specify one imagename to the command:

    rmimage <imagename>

Statelite Node Deployment

Statelite is an xCAT feature which allows you to have mostly stateless nodes (for ease of management), but tell xCAT that just a little bit of state should be kept in a few specific files or directories that are persistent for each node. If you would like to use this feature, refer to the [XCAT_Linux_Statelite] documentation.

Advanced features

Use the driver update disk:

Refer to [Using_Linux_Driver_Update_Disk].

Setup Kdump Service over Ethernet/HFI on diskless Linux (for xCAT 2.6 and higher)

Follow [Kdump over Ethernet or HFI for Linux diskless nodes] to define the diskless image object.

Generate rootimage for diskless/statelite

Follow XCAT_pLinux_Clusters#Stateless_node_deployment to generate the diskless rootimg . Follow XCAT_Linux_Statelite to generate the statelite image.

The Remaining Steps

Follow the documents including [xCAT_pLinux_Clusters] and [xCAT_Linux_Statelite] to setup the diskless/statelite image, and to make the specified noderange booting with the diskless/statelite image.

Additional configuration

After noderange booted up with the diskless/statelite image, add a dynamic ip range into networks table to the network used for compute node installation. This dynamic ip range should be large enough accommodate all of the nodes on the network. For example:

 #netname,net,mask,mgtifname,gateway,dhcpserver,tftpserver,nameservers,ntpservers,logservers,dynamicrange,nodehostname,ddnsdomain,vlanid,comments,disable
 "hfinet","20.0.0.0","255.0.0.0","hf0","20.7.4.1","20.7.4.1","20.7.4.1","20.7.4.1",,,"20.7.4.100-20.7.4.200",,,,,

References

Appendix A: Migrate your Management Node to a new Service Pack of Linux

If you need to migrate your xCAT Management Node with a new SP level of Linux, for example rhels6.1 to rhels6.2 you should as a precautionary measure:

Backup database and save critical files to be used if needed to reference or restore using xcatsnap. Move the xcatsnap log and *gz file off the Management Node.
Backup images and custom data in /install and move off the Management Node.
service xcatd stop
service xcatd stop on any service nodes
Migrate to the new SP level of Linux.
service xcatd start

If you have any Service Nodes:

Migrate to the new SP level of linux and reinstall the servicenode with xCAT following normal procedures.
service xcatd start

The document:

Setting_Up_a_Linux_xCAT_Mgmt_Node#Appendix_D:_Upgrade_your_Management_Node_to_a_new_Service_Pack_of_Linux
gives a sample procedure on how to update the management node or service nodes to a new service pack of Linux.

Appendix B: Install your Management Node to a new Release of Linux

First backup critical xCAT data to another server so it will not be loss during OS install.

Back up the xcat database using xcatsnap, important config files and other system config files for reference and for restore later. Prune some of the larger tables:
- tabprune eventlog -a
- tabprune auditlog -a
- tabprune isnm_perf -a (Power 775 only)
- tabprune isnm_perf_sum -a (Power 775 only)
Run xcatsnap ( will capture database, config files) and copy to another host. By default it will create in /tmp/xcatsnap two files, for example:
- xcatsnap.hpcrhmn.10110922.log
- xcatsnap.hpcrhmn.10110922.tar.gz
Back up from /install directory, all images, custom setup data that you want to save. and move to another server. xcatsnap will not backup images.

After the OS install:

Proceed to to setup the xCAT MN as a new xCAT MN using the instructions in this document.