Setting Up a Linux xCAT Mgmt Node
From xcat
Note: this document is no longer maintained, the same information is available in the xCAT iDataPlex Cluster Quick Start and the XCAT_iDataPlex_Advanced_Setup documents. They can be used to setup the Linux Management node, even if your cluster is not iDataPlex.
If you are using Flex servers follow xCAT system x support for IBM Flex or xCAT system p support for Linux on IBM Flex to set up the management node.
Hardware features required
Hardware requirements for your xCAT management node are dependent on your cluster size and configuration. A minimum requirement for an xCAT Management Node or Service Node that is dedicated to running xCAT to install a small cluster ( < 16 nodes) should have 4-6 Gigabytes of memory. A medium size cluster, 6-8 Gigabytes of memory; and a large cluster, 16 Gigabytes or more. Keeping swapping to a minimum should be a goal.
Install a Linux Management Node
To install the xCAT Management Node (MN), the following steps are recommended before installing the xCAT rpms:
- Install the MN with the OS. It is recommended that you install all rpms from the OS. If you choose not to, you should use yum/zypper to install xCAT to resolve all OS dependencies (e.g. dhcp, bind, expect, httpd, nfs-utils, vsftpd, etc.).
- Configure Cluster-Facing NICS
- Configure NTP
- Configure Hostname
- Configure DNS or some hostname resolution method.
- Setup basic /etc/hosts file
- Configure Ethernet switches
- Create a separate file system for /install
Documentation Convention: In order to distinguish different instructions for different operating systems or distributions, the notation [OS+distro+release] will be used before an operation to specify the applicable range of that operation. For example:
[RH] - applicable for all releases of RHEL and Fedora
[RHEL] - applicable for distro RHEL
[FEDORA] - applicable for distro Fedora
[SLES] - applicable for distro SLES
[SLES11] - applicable for distro SLES11
Install the OS of Management Node
To view the list of operating systems and hardware supported by xCAT, see: Hardware and OS Distos.
You must manually install the base Linux operating system on your xCAT management node using standard OS installation methodologies. xCAT does not provide specific instructions for doing this. We recommend that you install all rpm packages available for your operating system to provide you with all the necessary tools you may need to manage your cluster. If you choose not to do so, make sure you later use yum or zypper to install the xCAT rpms to ensure that all OS dependencies are automatically installed for you. Installing the xCAT rpms will be addressed in more detail below.
[RH] Disable SELinux
The RedHat SELinux (Secure Enterprise Linux) must be disabled on the xCAT 2 management node, various xCAT 2 functions will be affected, if the SELinux feature is enabled.
One method for disabling the SELinux is the following:
In the /etc/selinux/config file, set SELINUX=disabled. Reboot the management node.
[RH] Disable the Firewall
The management node provides many services to the cluster nodes, but the firewall on the management node can interfere with this. If your cluster is on a secure network, the easiest thing to do is to disable the firewall on the management node:
service iptables stop chkconfig iptables off
If disabling the firewall completely isn't an option, configure iptables to allow the following services on the NIC that faces the cluster: DHCP, TFTP, NFS, HTTP, FTP, DNS.
Setup Your Networks
When the xCAT rpms are installed on your management node, the install process will automatically scan and populate certain settings from your running configuration. Having all management networks configured to the operating system and activated ahead of time will aid in correct configuration in your xCAT database. All networks in your cluster must be defined in the xCAT database networks table before you can start to install cluster nodes. When the xCAT rpms are installed on your management node, they will run the xCAT makenetworks command to create an entry in the networks table for each of the network interfaces defined on the management node. Additional network configuration can be modified or added to the xCAT networks table manually later if needed.
Configure Cluster-Facing NICS
The cluster-facing NICs should be configured before installing your xCAT management node. There are a number of methods to configure the NICs such as updating Linux ifcfg-* files, using the Linux ifconfig command or the Linux Graphical User Interface. Here is an example of using ifcfg-* files to configure the NICs:
The location of the ifcfg-* files:
[RH]: /etc/sysconfig/network-scripts/ifcfg-*
[SLES]: /etc/sysconfig/network/ifcfg-*
An example of the content of ifcfg-* file:
DEVICE=eth1 ONBOOT=yes BOOTPROTO=static IPADDR=11.16.0.1 NETMASK=255.255.0.0
Prevent DHCP client from overwriting DNS configuration (Optional)
If the public facing NIC on your management node is configured by DHCP, you may want to set PEERDNS=no in the NIC's config file to prevent the dhclient from rewriting /etc/resolv.conf. This would be important if you will be configuring DNS on the management node (via makedns - covered later in this doc) and want the management node itself to use that DNS. In this case, set PEERDNS=no in each /etc/sysconfig/network-scripts/ifcfg-* file that has BOOTPROTO=dhcp.
On the other hand, if you want dhclient to configure /etc/resolv.conf on your management node, then don't set PEERDNS=no in the NIC config files.
Configure Hostname
The xCAT management node hostname must be configured before installing the xCAT rpms. The xCAT rpm installation will use the local hostname to automatically create SSL credentials and populate default information in your xCAT database. The hostname or its resolvable IP address will be used as the default master name in the xCAT site table. Therefore, this name needs to resolve to the IP address assigned to the NIC attached to the network that will be used to deploy and manage your cluster compute nodes. Short hostnames (no domain) are the norm for the Management Node and all cluster nodes. Names should never end in "-enx" for any x. For example, en0, en1.
If the management node hostname is already configured, this step can be skipped. The Linux command hostname or the Linux GUI can be used to configure the hostname. Editing the hostname configuration file /etc/sysconfig/network on RedHat systems or /etc/HOSTNAME on SuSE systems is another method.
[RH]: Edit the /etc/sysconfig/network and add entry 'HOSTNAME=(desired hostname)'.
[SLES]: Edit the /etc/HOSTNAME and add the proper hostname.
Configure DNS Resolution
/etc/resolv.conf should contain the DNS for management node itself.
search clusters.com nameserver 11.16.9.1
Set up basic hosts file
Ensure the /etc/hosts has the entries of localhost and management node itself.
127.0.0.1 localhost.localdomain localhost ::1 localhost6.localdomain6 localhost6 11.16.9.1 xcat20mn.clusters.com xcat20mn
Note: Name resolution across your cluster is required by xCAT. For now, setting up a simple /etc/hosts mechanism is enough to properly install your xCAT rpms and allow xCAT to set up initial defaults. Instructions will be provided later in the xCAT management node setup procedure to configure a full DNS name server for your cluster or to ensure that a complete /etc/hosts file gets distributed to all cluster nodes.
For more information on setting up name resolution in an xCAT Cluster: https://sourceforge.net/apps/mediawiki/xcat/index.php?title=Cluster_Name_Resolution
Setup the TimeZone
When installing the xCAT rpms on your management node, the local timezone setting will be set in the xCAT site table by default. This setting will later be used when installing service nodes or compute nodes. It is recommended that you set the correct timezone for your cluster on the management node before installing the xCAT rpms.
To set the timezone, first find the correct value to use. This will be a filename or subdir/filename in /usr/share/zoneinfo. Then:
vi /etc/sysconfig/clock
and set the appropriate value. For example:
[RH] ZONE="US/Eastern"
[SLES] TIMEZONE="America/New_York"
Also, you will need to use this for /etc/localtime:
ln -s /usr/share/zoneinfo/US/Eastern /etc/localtime
Enable ip forwarding (optional)
If the management node will be acting as the default gateway for the service nodes and compute nodes, the ip forwarding needs to be enabled on the management node, run the following commands to enable ip forwarding:
sed -i "s/^net.ipv4.ip_forward = .*/net.ipv4.ip_forward = 1/" /etc/sysctl.conf sysctl -e -p /etc/sysctl.conf
Clean up ssh keys (optional)
It is possible that the existing ssh keys under ~/.ssh are not in the right format that xCAT could recognize, it is a good practice to clean up the ssh public keys before installing xCAT, xCAT installation will re-create the ssh public keys.
cd ~/.ssh rm -rf id_rsa* rm -rf id_dsa*
A known problematic configuration scenario is that when the management node is converted from a xCAT compute node, the ~/.ssh/id_rsa.pub is not in the right format, you need to remove the ~/.ssh/id_rsa* before installing xCAT on this management node.
Restart Management Node (optional)
Though it is possible to restart the correct services for all settings except SELinux, the simplest step would be to reboot the management server at this point.
Configure Ethernet Switches (optional)
On system x and BladeCenter clusters, xCAT can use the ethernet switches for automatic hardware discovery and initial configuration. This is useful especially for large clusters. This requires that the user set up an IP address and basic SNMP functionality on the cluster ethernet switches. Allowing SNMP version 1 community string public read access will allow xCAT to communicate without further customization.
It is also recommended that spanning tree be set to portfast or edge-port for faster boot performance. Please see the relevant switch documentation as to how to configure these items.
Create a Separate File system for /install (optional)
It is not required, but recommended, that you create a separate file system for the /install directory on the Management Node. The size should be at least 30G to allow space for several install images.
Remove the package tftp-server
If you are a Power5 user or using xCAT 2.7 or above, then skip this section.
xCAT ships the atftp* package in its dependencies tarball. This package conflicts with the tftp-server rpm (or tftp rpm on SLES) that is installed by default with the OS.
In order to install xCAT successfully, you must first remove any tftp-server package that was previously installed.
[RH]: yum erase 'tftp-server.*'
[SLES]: zypper remove 'tftp.*'
or if tftp-server is required by other packages:
rpm -e --nodeps tftp-server # or tftp for SLES
Prepare For the Download and Install of xCAT on an MN That Has Direct Internet Access
xCAT can be downloaded and installed directly from sourceforge using YUM on RedHat systems or zypper on SuSE systems. This section explains how to set up the repo files to accomplish that.
For the xCAT core RPMs, you have 3 choices of what to download:
- The latest official release of the stable branch (recommended)
- The latest snapshot build of the stable branch - this includes recent bug fixes, but hasn't yet been fully tested
- The latest snapshot build of the development branch - this is the code the xCAT developers are currently working on. Use at your own risk.
The examples below show option #1. For the xCAT dependency RPMs, it is recommended that you always use the latest snap build available.
[RH] Setup Yum
Download repository files into the /etc/yum.repos.d:
cd /etc/yum.repos.d wget http://sourceforge.net/projects/xcat/files/yum/stable/xcat-core/xCAT-core.repo
Get the xCAT dep repo from http://sourceforge.net/projects/xcat/files/yum/xcat-dep/<release>/<arch>/xCAT-dep.repo . For example:
wget http://sourceforge.net/projects/xcat/files/yum/xcat-dep/rh6/ppc64/xCAT-dep.repo
[SLES] Setup Zypper
[SLES11]:
zypper ar -t rpm-md http://sourceforge.net/projects/xcat/files/yum/stable/xcat-core xCAT-core
zypper ar -t rpm-md http://sourceforge.net/projects/xcat/files/yum/xcat-dep/<release>/<arch> xCAT-dep
[SLES10.2+]:
zypper sa -t rpm-md http://sourceforge.net/projects/xcat/files/yum/stable/xcat-core xCAT-core
zypper sa -t rpm-md http://sourceforge.net/projects/xcat/files/yum/xcat-dep/<release>/<arch> xCAT-dep
Prepare For the Download and Install of xCAT For an MN That Does Not Have Direct Internet Access
Download and Unpack the Tarballs
Go to the Download xCAT site and download the level of xCAT tarball you want. Go to the xCAT Dependencies Download page and download the latest xCAT dependency tarball.
Copy the xCAT tarball files to the Management Node (MN) and untar them:
mkdir /root/xcat2
cd /root/xcat2
tar jxvf xcat-dep-*.tar.bz2
tar jxvf xcat-core-2.*.tar.bz2
OR
tar jxvf core-rpms-snap.tar.bz2
Setup YUM or zypper repositories for xCAT and Dependencies
Point yum/zypper to the local repositories for xCAT and its dependencies:
[RH]
cd /root/xcat2/xcat-dep/<release>/<arch> ./mklocalrepo.sh cd /root/xcat2/xcat-core ./mklocalrepo.sh
[SLES 11]:
zypper ar file:///root/xcat2/xcat-dep/sles11/<arch> xCAT-dep zypper ar file:///root/xcat2/xcat-core xcat-core
You can check a zypper repository using "zypper lr -d", or remove a zypper repository using "zypper rr".
[SLES 10.2+]:
zypper sa file:///root/xcat2/xcat-dep/sles10/<arch> xCAT-dep zypper sa file:///root/xcat2/xcat-core xcat-core
You can check a zypper repository using "zypper sl -d", or remove a zypper repository using "zypper sd".
Get the Requisite Packages From the Distro
xCAT depends on several packages that come from the Linux distro. Follow this section to create the repository of the OS on the Management Node.
[RHEL] Setup repository
To make the necessary RHEL RPM prereqs available to the xCAT install process, mount the RHEL CD/DVD or ISO and then create a repo file in /etc/yum.repos.d that points to it.
If you have the RHEL iso files:
If the RHEL distro only has one iso file:
Copy the iso file to any directory, such as /iso
mkdir /iso cp RHEL5.2-Server-20080430.0-ppc-DVD.iso /iso/
Mount the iso file to a directory such as /iso/rhels5.2
cd /iso mkdir /iso/rhels5.2 mount -o loop RHEL5.2-Server-20080430.0-ppc-DVD.iso /iso/rhels5.2
Create a YUM repository file, for example, rhel-dvd.repo, under directory /etc/yum.repos.d. The YUM repository contents should look like:
[rhe-5-server] name=RHEL 5 SERVER packages baseurl=file:///iso/rhels5.2/Server enabled=1 gpgcheck=1
Note: To make the YUM repository work persistently after management node reboot, the iso mount needs to be added into the /etc/fstab, or add the mount command into Linux startup scripts.
If the RHEL distro has more than one iso files
For each iso file:
- Loopback mount the iso file
- For the 1st iso file only, cd to the mounted directory and run: rpm --import RPM-GPG-KEY-redhat-release
- Copy the RPMs from the Server subdirectory of the mounted directory to a directory on your hard disk (for example /rhels5.3). You can put the RPMs from all of the iso files into the same directory.
- cd into the newly created RPM directory, install the createrepo RPM, and run: createrepo .
- Create a YUM repository file, for example, rhel-cd.repo, in directory /etc/yum.repos.d. The YUM repository contents should look like:
[rhel-5.3] name=RHEL 5.3 from directory baseurl=file:///rhels5.3 enabled=1 gpgcheck=1
For either case above:
change directory to where the RHEL CD image is and run
cd /iso/rhels5.2 rpm --import RPM-GPG-KEY-redhat-release
Check that the repo is set up correctly by looking for one rpm:
yum list screen
[SLES] Setup repository
If you have a SLES ISO:
mkdir /iso copy SLES11-DVD-ppc-GM-DVD1.iso to /iso/ mkdir /iso/1 cd /iso mount -o loop SLES11-DVD-ppc-GM-DVD1.iso 1 zypper ar [../../../iso/1 file:///iso/1] sles11
Check that the repo is set up correctly by looking for one rpm:
zypper search --match-exact -s screen
[FEDORA] Setup the repository
If your management node has access to the internet, you can simply create a file called /etc/yum.repos.d/fedora-internet.repo that contains:
[fedora-everything] name=Fedora $releasever - $basearch failovermethod=priority #baseurl=http://download.fedora.redhat.com/pub/fedora/linux/releases/$releasever/Everything /$basearch/os/ mirrorlist=http://mirrors.fedoraproject.org/mirrorlist?repo=fedora-$releasever&arch=$basearch enabled=1 gpgcheck=1 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora [../../../etc/pki/rpm-gpg/RPM-GPG-KEY file:///etc/pki/rpm-gpg/RPM-GPG-KEY]
Check that the repo is set up correctly by looking for one rpm:
yum list screen
If your management node does not have internet access, then download the iso of Fedora OS and setup the repository like [RHEL] Setup repository part.
For Fedora Operating System, xCAT still needs some packages which are not included in the installation iso, you need to download them from internet, copy them to the MN, and create a local YUM repository:
Note: The value of <releasever> in following link can be the release number of Fedora, it can be 8,9,12,13,14.
cd /root/xcat2/xcat-dep/fedoara/x86_64 export BASEURL=http://download.fedora.redhat.com/pub/fedora/linux/releases/<releasever>/Everything/x86_64/os/Packages/
[Fedora8]
wget $BASEURL/perl-Net-SNMP-5.2.0-1.fc8.1.noarch.rpm wget $BASEURL/perl-XML-Simple-2.17-1.fc8.noarch.rpm wget $BASEURL/perl-Crypt-DES-2.05-4.fc7.x86_64.rpm wget $BASEURL/net-snmp-perl-5.4.1-4.fc8.x86_64.rpm wget $BASEURL/ksh-20070628-1.1.fc8.x86_64.rpm wget $BASEURL/perl-IO-Socket-INET6-2.51-2.fc8.1.noarch.rpm wget $BASEURL/dhcp-3.0.6-10.fc8.x86_64.rpm wget $BASEURL/syslinux-3.36-7.fc8.x86_64.rpm wget $BASEURL/mtools-3.9.11-2.fc8.x86_64.rpm wget $BASEURL/expect-5.43.0-9.fc8.x86_64.rpm wget $BASEURL/perl-DBD-SQLite-1.12-2.fc8.1.x86_64.rpm wget $BASEURL/perl-Expect-1.20-1.fc8.1.noarch.rpm wget $BASEURL/perl-IO-Tty-1.07-2.fc8.1.x86_64.rpm wget $BASEURL/scsi-target-utils-0.0-1.20070803snap.fc8.x86_64.rpm wget $BASEURL/perl-Net-Telnet-3.03-5.1.noarch.rpm wget $BASEURL/perl-TimeDate-1.16.6.fc8.noarch.rpm wget $BASEURL/perl-DateTime-0.41-1.fc8.x86_64.rpm wget $BASEURL/perl-DateTime-Set-0.25-4.fc7.noarch.rpm wget $BASEURL/perl-MailTools-1.77-2.fc8.noarch.rpm wget $BASEURL/perl-Set-Infinite-0.61-3.fc7.noarch.rpm wget $BASEURL/perl-MIME-Lite-3.01-5.fc8.1.noarch.rpm wget $BASEURL/perl-version-0.7203-2.fc8.x86_64.rpm wget $BASEURL/perl-SOAP-Lite-0.68-5.fc8.noarch.rpm
[Fedora9]
You need to download rpm packages which are same with the Fedata8, but net-snmp-5.4.1-19.fc9 is needed for Fedora9.
[Fedora12]
wget $BASEURL/expect-5.43.0-19.fc12.i686.rpm wget $BASEURL/net-snmp-perl-5.4.2.1-18.fc12.x86_64.rpm wget $BASEURL/syslinux-3.75-4.fc12.x86_64.rpm wget $BASEURL/dhcp-4.1.0p1-12.fc12.x86_64.rpm wget $BASEURL/perl-XML-Simple-2.18-5.fc12.noarch.rpm wget $BASEURL/ksh-20090630-1.fc12.x86_64.rpm wget $BASEURL/perl-Test-Simple-0.92-82.fc12.x86_64.rpm wget $BASEURL/perl-Net-Telnet-3.03-9.fc12.noarch.rpm wget $BASEURL/ipmitool-1.8.11-4.fc12.x86_64.rpm wget $BASEURL/perl-Crypt-PasswdMD5-1.3-5.fc12.noarch.rpm wget $BASEURL/perl-DBD-SQLite-1.25-4.fc12.x86_64.rpm
[Fedora13]
wget $BASEURL/dhcp-4.1.1-15.fc13.x86_64.rpm wget $BASEURL/ipmitool-1.8.11-4.fc13.x86_64.rpm wget $BASEURL/syslinux-3.84-1.fc13.x86_64.rpm wget $BASEURL/ksh-20100309-3.fc13.x86_64.rpm wget $BASEURL/net-snmp-perl-5.5-12.fc13.x86_64.rpm wget $BASEURL/expect-5.43.0-19.fc12.i686.rpm wget $BASEURL/perl-Test-Simple-0.92-112.fc13.x86_64.rpm wget $BASEURL/perl-Crypt-PasswdMD5-1.3-6.fc13.noarch.rpm wget $BASEURL/perl-DBD-SQLite-1.27-3.fc13.x86_64.rpm
[Fedora14]
wget $BASEURL/expect-5.44.1.15-1.fc14.x86_64.rpm wget $BASEURL/ipmitool-1.8.11-5.fc14.x86_64.rpm wget $BASEURL/ksh-20100701-1.fc14.x86_64.rpm wget $BASEURL/net-snmp-perl-5.5-20.fc14.x86_64.rpm wget $BASEURL/perl-Crypt-SSLeay-0.58-1.fc14.x86_64.rpm wget $BASEURL/perl-DBD-SQLite-1.29-3.fc14.x86_64.rpm wget $BASEURL/perl-Test-Simple-0.94-2.fc14.noarch.rpm wget $BASEURL/perl-XML-LibXML-1.70-5.fc14.x86_64.rpm wget $BASEURL/perl-XML-NamespaceSupport-1.11-2.fc14.noarch.rpm wget $BASEURL/perl-XML-SAX-0.96-10.fc14.noarch.rpm wget $BASEURL/syslinux-4.02-3.fc14.x86_64.rpm
Since the dhcp packages of Fedara14 have issue to handle the omshell command, download the following two pakcages from fedora13 URL.
wget $BASEURL13/dhclient-4.1.1-15.fc13.x86_64.rpm wget $BASEURL13/dhcp-4.1.1-15.fc13.x86_64.rpm
At last, in this directory, run following command to create the local repository.
createrepo .
Install xCAT and Dependencies on the MN (yum/zypper)
[RH]:
yum clean metadata yum install xCAT # or yum install xCAT --nogpgcheck
[SLES]:
zypper install xCAT
To get to xCAT commands
source /etc/profile.d/xcat.sh
Additional Steps
[fedora15/redhat6.1]:
xCAT built a new conserver-xcat rpm package to replace the conserver rpm package for fedora15 and later. The 2.7 version of xCAT (and later) will automatically require conserver-xcat. For earlier versions of xCAT, uninstall conserver and install conserver-xcat manually before installing or upgrading xCAT:
rpm -e --force conserver yum install conserver-xcat
Install xCAT and Deps on the MN ( without yum/zypper)
If you chose not to use yum or zypper, then you will be managing all the dependencies that are needed for xCAT and it's dependency package. You may want to make sure the entire OS is installed on the Management Node to avoid having to search for missing dependencies. We recommend that you use yum/zipper for the install to avoid having to worry about missing dependencies.
You will need to download the xCAT rpms and dependencies as was described in Download and Install xCAT 2 For an MN That Does Not Have Internet Access.
A list of all the dependencies and xCAT rpms must be input to rpm. There are cross dependencies, xCAT requires some dependency rpms and some of the dependency rpms require xCAT rpms. For xCAT 2.4 and greater, atftp rpm is replaced with atftp-xcat rpm. You need to remove atftp rpm from the xcat-dep/<release>/<arch> directory before running the following commands.
cd /root/xcat2 rpm -Uvh xcat-dep/<release>/<arch>/*.rpm xcat-core/perl-xCAT*.rpm xcat-core/xCAT-2*.<arch>.rpm xcat-core/xCAT-client*.rpm xcat-core/xCAT-nbroot-core*.rpm xcat-core/xCAT-server*.rpm --replacepkgs
for example:
rpm -Uvh xcat-dep/fedora9/x86_64/*.rpm xcat-core/perl-xCAT*.rpm xcat-core/xCAT-2*.x86_64.rpm xcat-core/xCAT-client*.rpm xcat-core/xCAT-nbroot-core*.rpm xcat-core/xCAT-server*.rpm --replacepkgs
You can add these optional rpms to the list:
xcat-core/xCAT-rmc*.rpm xcat-core/xCAT-IBMhpc*.rpm xcat-core/xCAT-UI*.rpm
for rmc monitoring, hpc stack support and the web interface.
Do not install xCATsn*- this is only for a service node.
Note: The tftp client in the open firmware of Power 5 is only compatible with tftp-server instead of atftp-xcat/atftp rpm which is required by xCAT2. So you have to remove the atftp-xcat/atftp rpm with --nodeps flag first and then install the tftp-server. This is not required for Power6 or later.
The rpms installed on the MN, without the optional rpms, should look similar to this:
xCAT-nbkernel-x86_64-2.6.18_92-4.noarch perl-xCAT-2.4-snap201005031505.noarch xCAT-nbkernel-ppc64-2.6.18_92-4.noarch xCAT-nbroot-oss-x86_64-2.0-snap200801291344.noarch xCAT-nbroot-core-ppc64-2.4-snap201005031505.noarch xCAT-nbkernel-x86-2.6.18_92-4.noarch xCAT-client-2.4-snap201005031505.noarch xCAT-nbroot-core-ppc64-2.4-snap201004300946.noarch xCAT-nbroot-oss-x86-2.0-snap200804021050.noarch xCAT-nbroot-oss-ppc64-2.0-snap200801291320.noarch xCAT-nbroot-core-x86-2.4-snap201005031505.noarch xCAT-2.4-snap201005031505.x86_64 xCAT-nbroot-core-x86_64-2.2-snap200904010841.noarch xCAT-server-2.4-snap201005031505.noarch
Install/Update Service Node ( without yum/zypper)
You use the same process to as in the Install/Update xCAT and Deps on the MN ( without yum/zypper) for downloading and installing except one rpms changes.
Instead of the installing the xcat-core/xCAT-2*.rpm which is for the MN, you will install the xcat-core/xCATsn-2*.rpm for the service node.
So the command would look like the following:
cd /root/xcat2 rpm -Uvh xcat-dep/<release>/<arch>/*.rpm xcat-core/perl-xCAT*.rpm xcat-core/xCATsn-2*.rpm xcat-core/xCAT-client*.rpm xcat-core/xCAT-nbroot-core*.rpm xcat-core/xCAT-server*.rpm --replacepkgs
Test xCAT Installation
Add command to the path:
source /etc/profile.d/xcat.sh
Check to see the database is initialized:
tabdump site
The output should similar to the following:
key,value,comments,disable
"xcatdport","3001",,
"xcatiport","3002",,
"tftpdir","/tftpboot",,
"installdir","/install",,
.
.
.
Update xCAT Software at a Later Time
If you need to update the xCAT 2 rpms later:
Without Yum/Zypper
- If the management node does not have access to the internet: download the new version of xCAT Download and untar and install as before.
cd /root/xcat2
tar jxvf core-rpms-snap.tar.bz2
OR
tar jxvf xcat-core-2.x.tar.bz2
With Yum/Zypper
- If the management node has access to the internet, the yum/zypper command below will pull the updates directly from the xCAT site.
[RH]:
If you want to just update the xCAT rpms then run the below command. Note: this will not apply the changes that may have been made to the xCAT deps packages.
yum update '*xCAT*'
If you want to make all updates for xCAT, xCAT rpms and deps, run the following command. This command may also pick up additional OS updates.
yum update
[SLES]:
If you want to just update the xCAT rpms then run the below commands. Note: this will not apply the changes that may have been made to the xCAT deps packages.
zypper refresh zypper update -t package '*xCAT*'
If you want to make all updates for xCAT, xCAT rpms and deps, run the following commands. These command may also pick up additional OS updates.
zypper refresh zypper update
Note: If you have a service node stateless image in a hierarchical configuration , don't forget to update the image with the new xCAT rpms to keep the service node at the same level as the management Node.
Appendix A: Network Table Setup Example
And the following table shows all network IP addresses of the cluster:
| Machine name | IP Address | Alias |
|---|---|---|
| managementnode.pub.site.net | 10.0.12.53 | managementnode.site.net |
| managementnode.priv.site.net | 92.168.1.10 | |
| node01.pub.site.net | 10.0.12.61 | |
| node02.pub.site.net | 10.0.12.62 | |
| node01.infiniband.site.net | 10.0.6.236 | |
| node02.infiniband.site.net | 10.0.6.237 | |
| node01.10g.site.net | 10.0.17.14 | node01.site.net |
| node02.10g.site.net | 10.0.17.15 | node02.site.net |
| node01.priv.site.net | 192.168.1.21 | |
| node02.priv.site.net | 192.168.1.22 |
All networks in the cluster must be defined in the networks table which can be modified with the command chtab,chdef or with the command tabedit .
The xCAT 2 installation ran the command makenetworks which created the following entry:
# tabdump networks #netname,net,mask,mgtifname,gateway,dhcpserver,tftpserver,nameservers,ntpservers, - logservers,dynamicrange,nodehostname,comments,disable "10_0_12_0-255_255_255_0","192.168.1.0","255.255.255.0","eth1 - ",,,"192.168.1.10","10.0.12.10,10.0.17.10",,,,,, "192_168_1_0-255_255_255_0,"10.0.12.0","255.255.255.0","eth0 - ",,,"10.0.12.53","10.0.12.10,10.0.17.10",,,,,,
• Update the private network of this table as follow:
# chdef -t network -o "pvtnet" net=192.168.1.0 mask=255.255.255.0 mgtifname=eth0\ dhcpserver=192.168.1.10 tftpserver=192.168.1.10\ nameservers=10.0.12.10,10.0.17.10\ dynamicrange=192.168.1.21-192.168.1.22
• Disable the entry for the public network:
# chtab net=10.0.12.0 networks.disable=1 # tabdump networks #netname,net,mask,mgtifname,gateway,dhcpserver,tftpserver,nameservers,ntpservers, - logservers,dynamicrange,nodehostname,comments,disable "10_0_12_0-255_255_255_0","10.0.12.0","255.255.255.0","eth1",,,\ "10.0.12.53","10.0.12.10,10.0.17.10",,,,,,"1" "pvtnet","192.168.1.0","255.255.255.0","eth0",,\ "192.168.1.10","192.168.1.10","10.0.12.10,10.0.17.10",,,"192.168.1.21-192.168.1.22",,,,,,
Appendix B: Migrate your Management Node to a new Service Pack of Linux
If you need to migrate your xCAT Management Node with a new level of Linux, you should as a precautionary measure:
- Backup database and save critical files to be used if needed to reference or restore using xcatsnap. Move the xcatsnap log and *gz file off the Management Node.
- Backup images and custom data in /install and move off the Management Node.
- service xcatd stop
- service xcatd stop on any service nodes
- Migrate to the new level of Linux ( e.g. Redhat 6.0 to Redhat 6.1).
- service xcatd start
If you have any Service Nodes:
- Migrate to the new level of linux or reinstall with the new level of linux.
- service xcatd start
Note if you are using DB2 and moving from Redhat 6.0 and 6.1, you will have to upgrade from FixPack4 to FixPack5 of release 9.7 of DB2. Setting_Up_DB2_as_the_xCAT_DB#Appendix_B:Installing_DB2_fix_packs
Appendix C: Install your Management Node to a new Release of Linux
First backup critical xCAT data to another server so it will not be loss during OS install.
- Back up the xcat database using xcatsnap, important config files and other system config files for reference and for restore later. Prune some of the larger tables:
- tabprune eventlog -a
- tabprune auditlog -a
- tabprune isnm_perf -a (Power 775 only)
- tabprune isnm_perf_sum -a (Power 775 only)
- Run xcatsnap ( will capture database, config files) and copy to another host. By default it will create in /tmp/xcatsnap two files, for example:
- xcatsnap.hpcrhmn.10110922.log
- xcatsnap.hpcrhmn.10110922.tar.gz
- Back up from /install directory, all images, custom setup data that you want to save. and move to another server. xcatsnap will not backup images.
After the OS install:
- Proceed to to setup the xCAT MN as a new xCAT MN using the instructions in this document.
Appendix D: Upgrade your Management Node to a new Service Pack of Linux
Note: use this procedure at your own risk:
1) It is not fully tested.
2) It is not formally supported according to the Linux documentation.
3) This is only an example to upgrade RHEL 6 to RHEL 6.1, it might work for similar upgrade scenarios.
D.1. Prior to the upgrade
Do the following before the upgrade:
1. Stop xcatd, teal, cnmd and the db2 database on the management node. Service Nodes and compute nodes should be powered off per these instructions.
2. Make a list of your system's current packages for later reference:
rpm -qa --qf '%{NAME} %{VERSION}-%{RELEASE} %{ARCH}\n' > ~/old-pkglist.txt
3. (Optional, recommended by RedHat doc) Make a backup of any system configuration data and backup any other important data
tar czf /tmp/etc-`date +%F`.tar.gz /etc mv /tmp/etc-*.tar.gz /home/
4. Record the kernel version and release info of your current system
uname -a cat /etc/*release*
Then you can choose either upgrade from DVD (D.2) or upgrade use installer (D.3).
D.2 Upgrade from DVD
Details of the OS upgrade process can be obtained from <RedHat Enterprise Linux Installation Guide> -upgrade an Existing Installation http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/6/pdf/Installation_Guide/Red_Hat_Enterprise_Linux-6-Installation_Guide-en-US.pdf
Here are the basic steps:
1. Insert RedHat 6.1 installation DVD to DVD-ROM (or other boot device)
2. Reboot the system from a RedHat 6.1 installation DVD (or other boot media)
3. Enter the kernel option linux upgradeany at the boot: prompt.
4. Follow the dialog -Upgrade and Existing Installation ,then step by step
D.3 Upgrade using installer (good only for minor upgrade)
1. copycds /iso/RHEL6.1-20110510.1-Server-ppc64-DVD1.iso
2. Modify the yum repository file under /etc/yum.repos.d, for example:
Change from:
[rhe-6.0-server] name=RHEL 6.0 SERVER packages baseurl=file:///install/rhels6/ppc64/Server enabled=1 gpgcheck=1
to:
[rhe-6.1-server] name=RHEL 6.1 SERVER packages baseurl=file:///install/rhels6.1/ppc64/Server enabled=1 gpgcheck=1
3. To ensure that you only upgrade RedHat in this step, temporarily rename/move all other repo definition files in the /etc/yum.repos.d directory.
4. Perform upgrade
yum clean metadata yum -y update
During the update, I saw an error with updating glibc, not sure if it will impact the operating system:
Updating : glibc-2.12-1.25.el6.ppc64 15/420
Non-fatal POSTIN scriptlet failure in rpm package glibc-2.12-1.25.el6.ppc64
telinit:error.c:319: Assertion failed in nih_error_get: context_stack != NULL
/usr/sbin/glibc_post_upgrade: While trying to execute /sbin/telinit child terminated abnormally
warning: %post(glibc-2.12-1.25.el6.ppc64) scriptlet failed, exit status 118
4. Reboot the management node
D.4 After the upgrade
1. Verify the operating system has been updated
uname -a cat /etc/*release*
2. Check the operating system packages list, there might be some packages be uninstalled during the upgrade, reinstall any packages that were uninstalled during upgrade
awk '{print $1}' ~/old-pkglist.txt | sort | uniq > ~/old-pkgnames.txt
rpm -qa --qf '%{NAME}\n' | sort | uniq > ~/new-pkgnames.txt
diff -u ~/old-pkgnames.txt ~/new-pkgnames.txt | grep '^-' | sed 's/^-//' > /tmp/pkgs-toinstall.txt
If there is any package name in /tmp/pkgs-toinstall.txt, you need to install these packages either using yum or rpm command
3. Verify the network configuration files that are being used by xCAT are not changed, like:
/etc/dhcpd/dhcpd.conf /var/lib/dhcpd/dhcpd.leases /etc/ssh/sshd_config /etc/exports /etc/httpd/conf.d/xcat.conf
(any other files that need to be checked?)
4. Verify xcatd and some network services used by xCAT are still running, for example:
service xcatd status service named status service httpd status service xinetd status service ntpd status service vsftpd status
