Recent changes to Linux_AIX_mixed_cluster_configuration

Linux_AIX_mixed_cluster_configuration modified by Lissa Valletta

Lissa Valletta — Tue, 02 Sep 2014 14:12:38 -0000

--- v4
+++ v5
@@ -16,7 +16,9 @@

 * The Management must be Linux (one of the Linux OS's supported by xCAT).   It can be either system p or system x architecture.
 * You must have one or more AIX service nodes to install and support AIX compute nodes.  One of the AIX service nodes must be defined as the NIM prime
-* If you have both system x and system p linux nodes, the see also the instructions in [Mixed Cluster Support for SLES]
+* If you have both system x and system p linux nodes, the see also the instructions in 
+[Mixed_Cluster_Support_for_SLES]
+
 * Since you will be using hierarchy (service nodes), you must use either MySQL or Postgresql for the xCAT database.  DB2 is another option, but requires a license and is not supported on system x by xCAT. 
 * The first implementation of mixed AIX/Linux clusters only supports diskfull installs of the nodes(?). Stateless and statelite will be considered in future releases, if there is a requirement. 
 * Power 775 hardware is not support in a mixed cluster.
@@ -87,6 +89,7 @@
     postscripts=syslog,'''remoteshell,syncfiles''',aixremoteshell,syncfiles
     profile=Normal
     setupnameserver=no
+
 ~~~~

 As of xCAT2.8, the remoteshell postscript will call aixremoteshell, if running on an AIX node. You do not need to change the defaults for your AIX node.
@@ -107,6 +110,7 @@
     postscripts=syslog,remoteshell,syncfiles
     profile=Normal
     setupnameserver=no
+
 ~~~~

 ## Setup the remoteshell from management node to service node(s) 
@@ -148,7 +152,10 @@
 You will need to make sure that the Database rpms and the perl-DBD for the database are installed on the SN for the database you have chosen.   xCAT ships the rpms for both MySQL and PostgreSQL.   For the Service Node using MySQL, you will only need the perl-DBD-mysql*  installed and the /etc/xcat/cfgloc file copied from the Management Node.  For PostgreSQL install the rpms and also copy /etc/xcat/cfgloc to the SN.  No other setup is required on Service Node.  Follow the appropriate xCAT database doc for adding access to the database on the Management Node for each of your service nodes. 

 After you have done this, restart the xcatd on the Service Node and run 
+
+~~~~
  lsxcatd -a
+~~~~

 It should show you are using the correct database.  Run

Linux_AIX_mixed_cluster_configuration modified by Lissa Valletta

Lissa Valletta — Tue, 02 Sep 2014 14:10:28 -0000

--- v3
+++ v4
@@ -53,7 +53,9 @@

 The following documentation will help to define your nodes and service nodes to xCAT.  There will have to be additional setup for the mixed cluster.

-[XCAT_System_p_Hardware_Management]
+
+[XCAT_System_p_Hardware_Management_for_HMC_Managed_Systems]
+[xCAT_System_p_Hardware_Management_for_DFM_Managed_Systems]

 [Setting_Up_an_AIX_Hierarchical_Cluster]

Linux_AIX_mixed_cluster_configuration modified by Lissa Valletta

Lissa Valletta — Wed, 13 Aug 2014 13:20:03 -0000

--- v2
+++ v3
@@ -16,7 +16,7 @@

 * The Management must be Linux (one of the Linux OS's supported by xCAT).   It can be either system p or system x architecture.
 * You must have one or more AIX service nodes to install and support AIX compute nodes.  One of the AIX service nodes must be defined as the NIM prime
-* If you have both system x and system p linux nodes, the see also the instructions in [[Mixed Cluster Support for SLES]]
+* If you have both system x and system p linux nodes, the see also the instructions in [Mixed Cluster Support for SLES]
 * Since you will be using hierarchy (service nodes), you must use either MySQL or Postgresql for the xCAT database.  DB2 is another option, but requires a license and is not supported on system x by xCAT. 
 * The first implementation of mixed AIX/Linux clusters only supports diskfull installs of the nodes(?). Stateless and statelite will be considered in future releases, if there is a requirement. 
 * Power 775 hardware is not support in a mixed cluster.
@@ -26,7 +26,7 @@

 Refer to the  section titled “Install a Linux Management Node” in the following document for details on how to install a Linux management node. 

-[[XCAT_pLinux_Clusters]]
+[XCAT_pLinux_Clusters]

@@ -36,7 +36,7 @@

 Since the mixed cluster is actually a hierarchical structure, you will not be able to use the SQlite default database.  You can use MySQL, or Postgresql.  
 Use the documentation under "Hierarchy for Large Clusters", to help you setup your database and a Linux Hierarchical Cluster.
-[[XCAT_Documentation]]
+[XCAT_Documentation]
 MySQL is an easy one to use because it requires almost no code on the clients, which will be your service nodes in xCAT.

@@ -53,9 +53,9 @@

 The following documentation will help to define your nodes and service nodes to xCAT.  There will have to be additional setup for the mixed cluster.

-[[XCAT_System_p_Hardware_Management]]
+[XCAT_System_p_Hardware_Management]

-[[Setting_Up_an_AIX_Hierarchical_Cluster#Defining_Cluster_Nodes]]
+[Setting_Up_an_AIX_Hierarchical_Cluster]

 The “xcatdefaults” postscripts in the postscripts table are different on Linux management node and AIX management node.  In particular there are two xCAT scripts remoteshell for Linux and aixremoteshell for AIX. This is fixed in xCAT 2.8 ( you can define only remoteshell and it will take care of both Linux and AIX nodes).   We  are going to add the aixremoteshell script to the postscripts for the AIX nodes.   We are leaving the remoteshell script in for now ( it will fail on AIX).  Future releases of xCAT will merge these two and fix this problem. syncfiles only runs during install so will not actually be run during updatenode, thus having a duplicate will not be an issue.  You use updatenode -F to sync files. 
@@ -182,8 +182,8 @@

 After the AIX images are made available on the SN, all the subsequent steps are exactly the same with the homogeneous AIX cluster. Refer to the following documents for details on how to install or diskless boot the AIX compute nodes.

-[[XCAT_AIX_Diskless_Nodes]]
+[XCAT_AIX_Diskless_Nodes]

-[[XCAT_AIX_RTE_Diskfull_Nodes]]
+[XCAT_AIX_RTE_Diskfull_Nodes]

-[[XCAT_AIX_mksysb_Diskfull_Nodes]]
+[XCAT_AIX_mksysb_Diskfull_Nodes]

Linux_AIX_mixed_cluster_configuration modified by Lissa Valletta

Lissa Valletta — Wed, 13 Aug 2014 13:18:38 -0000

--- v1
+++ v2
@@ -22,7 +22,7 @@
 * Power 775 hardware is not support in a mixed cluster.

-== Install a Linux Management Node ==
+## Install a Linux Management Node 

 Refer to the  section titled “Install a Linux Management Node” in the following document for details on how to install a Linux management node. 

@@ -62,12 +62,15 @@

 Prior to xCAT 2.8, Your AIX Service node(s) should be created with these additional options:

+~~~~
  chdef <nodename> -p groups=aixservice setupnameserver=no os=AIX
  chtab node=aixservice postscripts.postscripts=aixremoteshell,syncfiles \  
  postscripts.postbootscripts=servicenode
+~~~~

 The results of lsdef where your nodename is substituted for snode1 below should look something like the following:

+~~~~
   lsdef snode1
  Object name: snode1
     cons=hmc
@@ -82,9 +85,12 @@
     postscripts=syslog,'''remoteshell,syncfiles''',aixremoteshell,syncfiles
     profile=Normal
     setupnameserver=no
+~~~~

 As of xCAT2.8, the remoteshell postscript will call aixremoteshell, if running on an AIX node. You do not need to change the defaults for your AIX node.
 So your lsdef of your nodename in xCAT 2.8 or later can look like the following:
+
+~~~~
   lsdef snode1
  Object name: snode1
     cons=hmc
@@ -99,11 +105,15 @@
     postscripts=syslog,remoteshell,syncfiles
     profile=Normal
     setupnameserver=no
+~~~~

 ## Setup the remoteshell from management node to service node(s) 

-At this point you need to setup ssh keys on the Service Node.  Note the service node must have a password assigned to root.  You will be prompted for it. 
+At this point you need to setup ssh keys on the Service Node.  Note the service node must have a password assigned to root.  You will be prompted for it.
+
+~~~~ 
  xdsh aixservice -K
+~~~~

@@ -111,8 +121,9 @@

 xCAT site table has an attribute “nimprime” to specify the hostname of the NIM Prime, use the following command to set the nimprime attribute in the site table.

-
- '''chdef -t site -o clustersite nimprime=<NIM Prime="" Host="" Name="">'''
+~~~~
+ chdef -t site -o clustersite nimprime=<NIM Prime="" Host="" Name="">
+~~~~

 Substitute the NIM Prime hostname as appropriate, our example was snode1

@@ -120,7 +131,9 @@

 For each AIX servicenode where the ip address below is the value of the site.master attribute. 

+~~~~
   chdef -t node <aixservicenode>  servicenode=192.168.5.93 xcatmaster=192.168.5.93
+~~~~

 ## Install xCAT Service Node packages on Service Node(s) 

@@ -136,16 +149,26 @@
  lsxcatd -a

 It should show you are using the correct database.  Run
+
+~~~~
  tabdump site
+~~~~
+
 You should get the site table from the MN.

 ## Setup the AIX Service Node 

 At this point, you should be able to created the required SSL credentials and copy the ssh keys to the Service Node by running on the MN:
+
+~~~~
  updatenode aixservice -k -V
+~~~~

 You can then run all your service node postscripts from the MN:
+
+~~~~
  updatenode aixservice -P
+~~~~

 To validate the service node setup, log in the service node and run tabdump site to see if it works.  Or run xdsh aixservice "/opt/xcat/sbin/tabdump site" from the MN.

Linux_AIX_mixed_cluster_configuration modified by Lissa Valletta

Lissa Valletta — Wed, 13 Aug 2014 13:15:44 -0000

Overview
- What is Supported and What is Not
Setup xCAT database on the MN
Install an AIX Service Node(s)
Define the AIX Service Node(s) as xCAT nodes
Setup the remoteshell from management node to service node(s)
Set nimprime attribute in the site table
Install xCAT Service Node packages on Service Node(s)
Setup the AIX Service Node
Make the AIX images available on the Service Node
Install or diskless boot the AIX compute nodes

Overview

'''Note: this support is under investigation and not ready for general use!'''

This “How-To” illustrates how to install AIX diskful and diskless compute nodes using Linux management node and AIX service node.

A cluster with both Linux nodes and AIX nodes is called “mixed cluster”. XCAT 2 only supports Linux management node to manage AIX service node and compute nodes for now. More scenarios will be supported in the future.

The process to setup mixed cluster is quite similar with the process to setup the normal xCAT 2 AIX cluster, only with some special steps. This “How-To” will not illustrates all the steps to setup a normal xCAT 2 AIX cluster, the link to the xCAT 2 AIX documentation is provided when necessary.

What is Supported and What is Not

The Management must be Linux (one of the Linux OS's supported by xCAT). It can be either system p or system x architecture.
You must have one or more AIX service nodes to install and support AIX compute nodes. One of the AIX service nodes must be defined as the NIM prime
If you have both system x and system p linux nodes, the see also the instructions in [[Mixed Cluster Support for SLES]]
Since you will be using hierarchy (service nodes), you must use either MySQL or Postgresql for the xCAT database. DB2 is another option, but requires a license and is not supported on system x by xCAT.
The first implementation of mixed AIX/Linux clusters only supports diskfull installs of the nodes(?). Stateless and statelite will be considered in future releases, if there is a requirement.
Power 775 hardware is not support in a mixed cluster.

== Install a Linux Management Node ==

Refer to the section titled “Install a Linux Management Node” in the following document for details on how to install a Linux management node.

[[XCAT_pLinux_Clusters]]

Note: When the Linux management node is running on a system x server, the perl-IO-Stty needs to be installed on the management node to perform hardware control operations on the the system p service nodes and compute nodes.

Setup xCAT database on the MN

Since the mixed cluster is actually a hierarchical structure, you will not be able to use the SQlite default database. You can use MySQL, or Postgresql.
Use the documentation under "Hierarchy for Large Clusters", to help you setup your database and a Linux Hierarchical Cluster.
[[XCAT_Documentation]]
MySQL is an easy one to use because it requires almost no code on the clients, which will be your service nodes in xCAT.

Install an AIX Service Node(s)

To configure a mixed cluster with Linux management node and AIX compute nodes, there must be one or more AIX service node machines to act as the AIX NIM Prime. All the necessary NIM resources will be created on the NIM Prime/Service Node (SN) and then the AIX service nodes provides the NIM resources to the compute nodes. In the current release, xCAT 2 only supports the scenario that the NIM Prime and service node is on the same machine.

The AIX NIM Prime SN can be installed through various ways such as AIX CD/DVD media or NIM. The SSL and SSH filesets are needed by xCAT, so please make sure the SSL and SSH filesets are installed on the SN. You can refer to the two AIX installp bundle files xCATaixSSL.bnd and xCATaixSSH.bnd , which are shipped with the xCAT image core-aix-<version>.tar.gz file, for the detailed packages list.

If the AIX servicenode is not install with xCAT, then run updatenode <servicenode> -s to updatenode the /etc/xcatinfo file on the node.

Define the AIX Service Node(s) as xCAT nodes

The following documentation will help to define your nodes and service nodes to xCAT. There will have to be additional setup for the mixed cluster.

[[XCAT_System_p_Hardware_Management]]

[[Setting_Up_an_AIX_Hierarchical_Cluster#Defining_Cluster_Nodes]]

The “xcatdefaults” postscripts in the postscripts table are different on Linux management node and AIX management node. In particular there are two xCAT scripts remoteshell for Linux and aixremoteshell for AIX. This is fixed in xCAT 2.8 ( you can define only remoteshell and it will take care of both Linux and AIX nodes). We are going to add the aixremoteshell script to the postscripts for the AIX nodes. We are leaving the remoteshell script in for now ( it will fail on AIX). Future releases of xCAT will merge these two and fix this problem. syncfiles only runs during install so will not actually be run during updatenode, thus having a duplicate will not be an issue. You use updatenode -F to sync files.

Prior to xCAT 2.8, Your AIX Service node(s) should be created with these additional options:

chdef <nodename> -p groups=aixservice setupnameserver=no os=AIX
chtab node=aixservice postscripts.postscripts=aixremoteshell,syncfiles \
postscripts.postbootscripts=servicenode

The results of lsdef where your nodename is substituted for snode1 below should look something like the following:

lsdef snode1
Object name: snode1
cons=hmc
groups=groups=lpar,all,aixservice
hcp=ac0131a
id=10
mgt=hmc
nodetype=lpar,osi
os=AIX
parent=p1vsp01
postbootscripts=otherpkgs,servicenode
postscripts=syslog,'''remoteshell,syncfiles''',aixremoteshell,syncfiles
profile=Normal
setupnameserver=no

As of xCAT2.8, the remoteshell postscript will call aixremoteshell, if running on an AIX node. You do not need to change the defaults for your AIX node.
So your lsdef of your nodename in xCAT 2.8 or later can look like the following:
lsdef snode1
Object name: snode1
cons=hmc
groups=groups=lpar,all,aixservice
hcp=ac0131a
id=10
mgt=hmc
nodetype=lpar,osi
os=AIX
parent=p1vsp01
postbootscripts=otherpkgs,servicenode
postscripts=syslog,remoteshell,syncfiles
profile=Normal
setupnameserver=no

Setup the remoteshell from management node to service node(s)

At this point you need to setup ssh keys on the Service Node. Note the service node must have a password assigned to root. You will be prompted for it.
xdsh aixservice -K

Set nimprime attribute in the site table

xCAT site table has an attribute “nimprime” to specify the hostname of the NIM Prime, use the following command to set the nimprime attribute in the site table.

'''chdef -t site -o clustersite nimprime=<NIM Prime="" Host="" Name="">'''

Substitute the NIM Prime hostname as appropriate, our example was snode1

When setting the nimprime, you must explicitly your AIX service nodes to point to the site.master as their service nodes

For each AIX servicenode where the ip address below is the value of the site.master attribute.

chdef -t node <aixservicenode> servicenode=192.168.5.93 xcatmaster=192.168.5.93

Install xCAT Service Node packages on Service Node(s)

Various ways can be used to install the xCAT service node packages on the AIX service node(s). Note these are the same as the Management Node packages except you will be installing the xCATSN-2.x.x package, '''not''' the xCAT-2.x.x package. For example, you can copy all the xCAT service node packages to the AIX service node(s) and install these packages by running geninstall/rpm command on the service node(s) separately or by running xdsh command against the service node(s) to install the service node packages on all the service node(s) at same time; the updatenode software maintenance function is another option.

Refer the AIX installp bundle file xCATaixSN2.bnd, which is shipped with the xCAT image core-aix-<version>.tar.gz file, for the detailed service node packages list.

You will need to make sure that the Database rpms and the perl-DBD for the database are installed on the SN for the database you have chosen. xCAT ships the rpms for both MySQL and PostgreSQL. For the Service Node using MySQL, you will only need the perl-DBD-mysql* installed and the /etc/xcat/cfgloc file copied from the Management Node. For PostgreSQL install the rpms and also copy /etc/xcat/cfgloc to the SN. No other setup is required on Service Node. Follow the appropriate xCAT database doc for adding access to the database on the Management Node for each of your service nodes.

After you have done this, restart the xcatd on the Service Node and run
lsxcatd -a

It should show you are using the correct database. Run
tabdump site
You should get the site table from the MN.

Setup the AIX Service Node

At this point, you should be able to created the required SSL credentials and copy the ssh keys to the Service Node by running on the MN:
updatenode aixservice -k -V

You can then run all your service node postscripts from the MN:
updatenode aixservice -P

To validate the service node setup, log in the service node and run tabdump site to see if it works. Or run xdsh aixservice "/opt/xcat/sbin/tabdump site" from the MN.

Make the AIX images available on the Service Node

All the NIM resources will be created on the Service Node, so the AIX images must be made available on the SN. The AIX images could be the AIX product media, a directory containing the AIX images, or the name of an existing NIM lpp_source resource. The full path of the AIX images on the SN will be used by the commands mknimimage and mkdsklsnode.

Install or diskless boot the AIX compute nodes

After the AIX images are made available on the SN, all the subsequent steps are exactly the same with the homogeneous AIX cluster. Refer to the following documents for details on how to install or diskless boot the AIX compute nodes.

[[XCAT_AIX_Diskless_Nodes]]

[[XCAT_AIX_RTE_Diskfull_Nodes]]

[[XCAT_AIX_mksysb_Diskfull_Nodes]]