Cluster_Name_Resolution
There is a newer version of this page. You can find it here.
- Cluster Name Resolution
- Support for multiple network domain names (work in progress)
- Specifying additional network interfaces for cluster nodes (work in progress)
- Limited support for user application network interfaces (work in progress)
Cluster Name Resolution
Setting up name resolution and having the nodes resolve to IP addresses is required in xCAT clusters.
You can use a simple /etc/hosts mechanism or you can optionally set up a DNS name server.
If you do not set up DNS you will have to distribute new versions of the /etc/hosts file to all the cluster nodes whenever you add new nodes to the cluster.
NOTE: This documentation only applies to the xCAT makedns command using the ddns.pm plugin, which is the new default dnshandler in xCAT 2.6.2 and above.
NOTE: The ddns.pm plugin is based on named9/bind9, can not support named8/bind8 due to syntax difference. For example, if you are using AIX61, named points to named8 by default, you need to link named to named9 before makedns is issued.
NOTE: makedns command will start named/bind service automatically, and also enable named/bind on boot. If you decide not to use named/bind after next boot, you can turn it off by:
_**rmitab named # AIX**_
_**chkconfig named off # Linux except ubuntu**_
_**chkconfig bind9 off # ubuntu**_
To set up the management node as the DNS name server you must set the "domain", "nameservers" and "forwarders" attributes in the xCAT "site" definition.
For example, if the cluster domain is "mycluster.com", the IP address of the management node, (as known by the cluster nodes), is "100.0.0.41" and the site DNS servers are "9.14.8.1,9.14.8.2" then you would run the following command.
_**chdef -t site domain= mycluster.com nameservers= 100.0.0.41 forwarders= 9.14.8.1,9.14.8.2**_
Create an /etc/resolv.conf file on the management node
Edit "/etc/resolv.conf" to contain the cluster domain and nameserver values you set in the site table above. For example:
_**
search mycluster.com
nameserver 100.0.0.41**_
Note: If you are using xCAT 2.7.0 or beyond, this step is not necessary.
Populate the /etc/hosts File
All of the nodes should be added to the /etc/hosts file on the xCAT management node. You can either edit the /etc/hosts file by hand, or use makehosts.
If you edit the file by hand, it should look similar to:
_**
127.0.0.1 localhost localhost.localdomain
50.1.2.3 mgmtnode-public mgmtnode-public.cluster.com 10.0.0.100 mgmtnode mgmtnode.cluster.com 10.0.0.1 node1 node1.cluster.com 10.0.0.2 node2 node2.cluster.com**_
Support for multiple network domain names is available starting in the xCAT 2.8 release. See the section later in this document titled "Support for multiple network domain names" for more information on using multiple network domains in an xCAT cluster environment.
If your node names and IP addresses follow a regular pattern, you can easily populate /etc/hosts by putting a regular expression in the xCAT hosts table and then running makehosts. To do this, you need to first create an initial definition of the nodes in the database, if you haven't done that already:
_**
mkdef node[01-80] groups=compute,all
**_
**** Next, put a regular expression in the hosts table. The following example will associate IP address 10.0.0.1 with node1, 10.0.0.2 with node2, etc:
_**
chdef -t group -o compute ip='|node(\d+)|10.0.0.($1+0)|'
**_
Then run
_**
makehosts compute
**_
and the following entries will be added to /etc/hosts:
_**
10.0.0.1 node01 node01.cluster.com
10.0.0.2 node02 node02.cluster.com 10.0.0.3 node03 node03.cluster.com ...**_
Starting with xCAT release 2.8 you may also specify additional network interface information in the xCAT node or group definitions.
This information is used by the makehosts command to add the additional interface hostnames to the /etc/hosts file. It is also used by xCAT adapter configuration postscripts to automatically config the additional network interfaces on the node. See the section later in this document titled " " for more information.
For an explanation of the regular expressions, see the xcatdb man page.
Note that it is a convention of xCAT that for Linux systems the short hostname is the primary hostname for the node, and the long hostname is an alias.
On AIX the order is typically reversed. To have the long hostname be the primary hostname, you can use the -l option on the makehosts command.
Create xCAT Network Definitions
An xCAT "network" definition must be created for each management network used in the cluster.
When you installed xCAT, it created network definitions for the networks that the management node is connected to (i.e. has NICs configured for). If the cluster-facing NICs were not configured when xCAT was installed, or if there are more cluster networks that are only available via the service nodes, you must create new network definitions now.
You will need a name for the network and values for the following attributes.
**net** The network address.
**mask** The network mask.
**gateway** The network gateway.
You can use the xCAT makenetworks command to gather cluster network information and create xCAT network definitions. See the makenetworks for details.
You can also use the xCAT mkdef command to manually define the network.
For example you could run a command similar to the following.
_**
mkdef -t network -o clusternet net=10.0.0.0 mask=255.0.0.0 gateway=10.0.0.254
**_
Note: The makedns command (mentioned below) will only add nodes into the DNS configuration if the network for the node is defined.
Using DNS
DNS Choices
There are many different ways to configure DNS name resolution in your cluster. Some of the common choices are:
- In a basic (non-hierarchical) cluster, point all nodes to a DNS running on the management node.
- In a basic (non-hierarchical) cluster, point all nodes to an external DNS running at your site.
- In a hierarchical cluster, point all compute nodes to their service node
Option #1: Management Node DNS
This is the most common set up. In this configuration, a DNS running on the management node handles all name resolution requests for cluster node names. A separate DNS in your site handles requests for non-cluster hostnames.
To set up the management node as the DNS name server you must set the "domain", "nameservers" and "forwarders" attributes in the xCAT "site" definition.
Set site "forwarders" to DNS servers outside of the cluster that can resolve non-cluster hostnames. The management node DNS will automatically forward requests that it can't handle to these name servers.
For example, if the cluster domain is "mycluster.com", the IP address of the management node, (as known by the cluster nodes), is "100.0.0.41" and the site DNS servers are "50.1.2.254,50.1.3.254" then you would run the following command.
**_chdef -t site domain=mycluster.com nameservers=100.0.0.41 forwarders=_50.1.2.254,50.1.3.254''**__
Once /etc/hosts is populated with all of the nodes' hostnames and IP addresses, configure DNS on the management node and start it:
_** makedns -n **_
_** chkconfig named on # linux **_
_** startsrc -s named # AIX**_
The resolv.conf files for the compute nodes will be created automatically using the "domain" and "nameservers" values set in the site definition.
Note: It is also possible to set the "domain" and "nameservers" values in the xCAT network definitions. For AIX nodes it is also possible to use a NIM resolv_conf resource.
If you add nodes or change node names or IP addresses later on, rerun makedns which will automatically restart named.
To verify the DNS service on management node is working or not:
_** nslookup <host> <mn's ip> **_
For example:
_** nslookup node1 100.0.0.41**_
Option #2: DNS Outside of the Cluster
If you already have a DNS on your site network and you want to use that for your cluster node names too, you can point all of the nodes to it. You must ensure that your nodes have IP connectivity to the DNS, and you must manually configure your DNS with the node hostnames and IP addresses.
Set the site "nameservers" value to the IP address of the existing name server.
_**chdef -t site nameservers=50.1.2.254**_
In this case each node will have an /etc/resolv.conf file that contains the "domain" and "namservers" values from the site definition.
Option #3: Service Node DNS
When you have service nodes, the recommended configuration is to run DNS on the management node and all of the service nodes. The DNS on the management node is the only one configured with all of the node hostname/IP address pairs. The DNS servers on the service nodes are simply forwarding/caching servers.
Set site "forwarders" value to DNS servers outside of the cluster that can resolve non-cluster hostnames. The management node DNS will automatically forward requests that it can't handle to these name servers:
_**chdef -t site forwarders=50.1.2.254,50.1.3.254**_
Note: only the DNS on the management node will use the forwarders setting. The DNS servers on the service nodes will always forward requests to the management node.
Once /etc/hosts is populated with all of the nodes' hostnames and IP addresses, configure DNS on the management node and start it:
_** makedns -n **_
_** chkconfig named on # linux **_
_** startsrc -s named # AIX**_
In this case, when the /etc/resolv.conf files for the compute nodes are created the value of the nameserver must be the name of the nodes' server - as known by the node.
To do this the "nameservers" attribute should be set to "<xcatmaster>".
_**
chdef -t site nameservers="<xcatmaster>"
**_
The "<xcatmaster>" keyword will be interpreted as the value of the "xcatmaster" attribute of the node definition. The "xcatmaster" value for a node is the name of it's server as known by the node. This would be either the cluster-facing name of the service node or the name of the management node.
Note: for Linux, the site "nameservers" value must be set to "<xcatmaster>" before you run makedhcp.
Make sure that the DNS service on the service nodes will be set up by xCAT.
Assuming you have all of your service nodes in a group called "service" you could run a command similar to the following.
_**chdef -t group service setupnameserver=1**_
For Linux systems, make sure DHCP is set up on the service nodes.
_**chdef -t group service setupdhcp=1**_
If you have not yet installed or diskless booted your service nodes, xCAT will take care of configuring and starting DNS on the service nodes at that time. If the service nodes are already running, restarting xcatd on them will cause xCAT to recognize the above setting and configure/start DNS:
_**xdsh service 'service xcatd restart' # linux**_
_**xdsh service 'restartxcatd' # AIX**_
If you add nodes or change node names or IP addresses later on, rerun makedns. The DNS on the service nodes will automatically pick up the new information.
Creating node resolv.conf files on AIX - Overview
The resolv.conf files could be created manually or you could use the xCAT support to automate the creation of the files when the nodes are installed.
This section provides an overview of the various ways you can set up resolv.conf files in an AIX cluster.
Providing "domain" and "nameservers" values
The resolv.conf file requires a "domain" value and a "nameservers" value. When using xCAT on AIX you must provide this information in one of three ways:
1) Create a NIM resolv_conf resource and include it in your xCAT osimage definition. To do this you must create a NIM reolv_conf resource on the xCAT management node and include it in the xCAT osimage definition you will use to install the cluster nodes.
2) Add values for the "domain" and "nameservers" attributes to the xCAT "site" definition. When setting up a cluster the xCAT "site" definition is used to save some basic cluster-wide information. You can add the "domain" and "nameservers" values to the "site" definition using the xCAT chdef command. For example:
_chdef -t site domain=mycluster.com nameservers=30.1.0.102_
3) Add values for the "domain" and "nameservers" attributes to the xCAT "network" definitions. An xCAT "network" definition must be created for each network that is used for cluster management. You can add the "domain" and "nameservers" values to the network definitions using the chdef command. For example:
_chdef -t network -o cluster_net domain=mycluster.com nameservers=30.1.0.102_
When xCAT is determining the resolv.conf file for a node it will use the following priority order:
- The user provided resolv_conf resource is used if provided.
- If there is no resolv_conf resource then the information from the network definition, that corresponds to the node, will be used. (I.e. If the node is located on the "cluster_net" network then the "domain" and "nameservers" values from that network definition will be used.
- If there is no resolv_conf resource AND the network definition does not contain the "domain" and "nameservers" values then the "site" definition will be used.
- If the "domain" and "nameservers" values are not provided in any of the above then no resolv.conf file will be created on the nodes.
When setting the value for the "nameservers" attribute you may either use a comma-separated list of server IP addresses OR the keyword "<xcatmaster>". The "<xcatmaster>" keyword will be interpreted as the value of the "xcatmaster" attribute of the node definition. The "xcatmaster" value for a node is the name of it's server as known by the node. This would be either the name of the service node or the name of the management node.
Using /etc/hosts throughout the cluster
Note: currently this option only works easily on AIX. To make it work on linux, you have to also manually change /etc/nsswitch.conf .
If you choose to use a fully populated /etc/hosts file on every node, you must ensure that the file gets updated throughout the cluster every time a node is added or a hostname or IP address changes.
**NOTE:** You should also remove the "nameservers" setting from the site definition and from any "network" definiton so that xCAT will not generate an /etc/resolv.conf file for the nodes. For example:
_****_
chdef -t site nameservers=
There are several methods you can use to update the /etc/hosts files on the nodes.
1) Use the xCAT xdcp command to copy the new files to the nodes.
See [xdcp](http://xcat.sourceforge.net/man1/xdcp.1.html) for details.
In this case the new /etc/hosts file would not be available during the boot of the node.
2) Use the xCAT sync files support.
For example, you could create a synclist file containing the following entry:
_****_
/etc/hosts -> /etc/hosts
The file can be put anywhere, but let's assume you name it /install/custom/compute-image/synclist .
Make sure you have an OS image object in the xCAT database associated with your nodes:
_****_
mkdef -t osimage compute-image synclists=/install/custom/compute-image/synclist
_****_
chdef -t group compute provmethod=compute-image
Each time you install or diskless boot a compute node, xCAT will automatically sync the /etc/hosts file to the node. If you make changes to /etc/hosts while the nodes are running, you must push those changes to the nodes:
_****_
updatenode compute -F
In this case the new /etc/hosts file would not be available during the boot of the node.
3) Use the xCAT statelite support for diskless nodes.
If all of your nodes are statelite nodes you can add /etc/hosts to the [litefile](http://xcat.sourceforge.net/man5/litefile.5.html) table with the "ro" option and add the place it can be mounted from to the [litetree](http://xcat.sourceforge.net/man5/litetree.5.html) table.
For Linux systems see [Linux statelite doc](XCAT_Linux_Statelite) for details on using the statelite support.
For AIX systems refer to the documentation on AIX Diskless Nodes:
[XCAT_AIX_Diskless_Nodes] for details on using the statelite support. See the section titled "Provide a read-only configuration file" for an example of how you could set up the /etc/hosts file.
In this case the new /etc/hosts file will be available during the boot of the node.
Support for multiple network domain names (work in progress)
Note: This support is available starting with the xCAT 2.8 release.
xCAT supports the use of multiple network domain names within the xCAT cluster environment.
If you wish to use only one network domain name then you can set the "domain" attribute of the xCAT "site" definition.
ex. "chdef -t site domain=cluster.com"
If you wish to use multiple domains you must set the "domain" attribute of the corresponding xCAT "network" defintion.
ex. "chdef -t network -o clusternet1 domain=public.com"
The multiple domain support does not include support for multiple domains per subnet.
To determine the "domain" of a cluster node the xCAT code will check the network definition corresponding to the node ip address and then get the "domain" name from the network definition. If the "doamin" is not set in the network definition then the code will use the "domain" value set in the xCAT "site" definition.
Specifying additional network interfaces for cluster nodes (work in progress)
Note: This support is available starting with the xCAT 2.8 release.
You can specify additional interface information as part of an xCAT node definition by using the following attributes:
nicips - Comma-separated list of IP addresses per additional interface (NIC). (Required)
nichostnamesuffixes - Comma-separated list of hostname suffixes per NIC. (Required)
nictypes - Comma-separated list of NIC types per NIC. (Optional)
niccustomscripts - Comma-separated list of custom configuration scripts per NIC. (Optional)
nicnetworks - Comma-separated list of xCAT network definition names corresponding to each NIC. (Optional)
For a description of the nic* attributes see the nics man page.
To set and/or modify the nic information for a node you can either modify the node or group definition using the chdef* command, or you can modify the xCAT database table directly using the "tabedit nics" command.
This information for additional interfaces is used for such things as populating the /etc/hosts file with the extra interfaces (makehosts) and providing xCAT adapter configuration scripts with the information to automatically configure the additional interfaces.
The details for how to add the additional information is described below. As you will see, entering this information manually can be tedious and error prone. This support is primarily targetted to be used in conjunction with other IBM products that have tools to fill in this information in an automated way.
In some cases, when you need to add this information manually, you may be able to use regular expressions to simplify the process. See the "Examples" section below for details.
Syntax of nic attribute values.
- Each nic* value can include information for multiple additional interfaces separated by commas.
- The information for each interface must be in the form "<nicname>!<nicinfo>".
- To provide multiple values for an interface you must use a "|" delimiter.
nicips example:
chdef -t node -o compute02 nicips="eth0!11.10.1.2,eth2!80.0.0.2|fd55::214:5eff:fe15:849b"
NOTE: The interface (eth1), that the compute02 IP is configured on, is not included in the list of additional nics. Although adding it to the list of nics would do no harm.
This "nicips" value indicates that there are two additional interfaces to be configured on node compute02, eth0 and eth2. The eth0 interface will get the IP address "11.10.1.2". The eth2 interface will get two IP addresses, 80.0.0.2 and fd55::214:5eff:fe15:849b.
nichostnamesuffixes example
chdef -t node -o compute02 nichostnamesuffixes="eth0!-eth0,eth2!-eth2|-eth2-ipv6"
This value indicates that the hostname for eth0 should be "compute02-eth0". For eth2 we had two IP addresses so now we need two suffixes. The hostnames for eth2 will be "compute02-eth2" and "compute02-eth2-ipv6". The IP for "compute02-eth2" will be "80.0.0.2" and the IP for "compute02-eth2-ipv6" will be "fd55::214:5eff:fe15:849b".
nictypes example
chdef -t node -o compute02 nictypes="eth0!ethernet,eth2!ethernet|ethernet"
This value indicates that all the nics are ethernet. The valid "nictypes" values are: "ethernet", "infiniband", and "bmc".
niccustomscripts example
chdef -t node -o compute02 niccustomscripts="eth0!cfgeth,eth2!cfgeth|cfgeth"
In this example "cfgeth" is the name of an adapter configuration postscript to be used to configure the interface.
nicnetworks example
chdef -t node -o compute02 nicnetworks="eth0!clstrnet11,eth2!clstrnet80|clstrnet-ipv6"
In this example we are saying that the IP address of eth0 (ie. compute02-eth0 -> 11.10.1.2) is part of the xCAT network definition named "clstrnet11". "compute02-eth2" is in network "clstrnet80" and "compute02-eth2-ipv6" is in "clstrnet-ipv6".
If the xCAT network definition name is not provided then the xCAT code will attempt to match the interface IP to one of the xCAT network definitions.
An xCAT network definition must be created for all networks being used in the xCAT cluster environment.
Examples
Add additional NIC information for a single cluster node
In this example we assume that we have designated that node "compute01" get IP address "60.0.0.1" which will be configured on interface "eth0". This will be the xCAT management interface for the node. In addition to the management interface we also wish to include information for the "eth1" interface on node "compute01". To do this we must set the additional nic information for this node. For example:
chdef -t node -o compute01 nicips="eth1!80.0.0.1" nichostnamesuffixes="eth1!-eth1" nictypes="eth1!ethernet"
This information will be used to configure the "eth1" interface, in addition to the management interface (eth0), during the boot of the node.
Also, if you were to run "makehosts compute01" at this point you would see something like the following entries added to the /etc/hosts" file.
60.0.0.1 compute01 compute01.cluster60.com
80.0.0.1 compute01-eth1 compute01-eth1.cluster80.com
NOTE: If you specify the same IP address for a nic as you did for the management interface then the nic hostname will be considered an alias of the xCAT node hostname. For example, if you specified "60.0.0.1" for the eth1 "nicips" value then the /etc/hosts entry would be:
60.0.0.1 compute01 compute01.cluster60.com compute01-eth1
Add additional NIC information for all cluster computational nodes
In this example we'd like to configure additional eth1 interfaces on each computational node in the cluster.
The basic approach will be to create an xCAT node group containing all the nodes and then use a regular expression to determine the actual nicips to use for each node.
For this technique to work you must set up the hostnames and IP address to a have a regular pattern. For more information on using regular expression in the xCAT database see the xcatdb man page.
In the following example, the xCAT node group "compute" was defined to include all the compute nodes: compute01, compute02, compute03 etc. These hostnames/IPs will be mapped to the eth0 interfaces.
For the eth1 interfaces on these nodes we'd like to have compute01-eth1 map to 80.0.0.1, and compute02-eth1 to map to 80.0.0.2 etc.
To do this we could defined the "compute" group attributes as follows:
chdef -t group -o compute nicips='|\D+(\d+)|eth1!80.0.0.($1+0)|' nichostnamesuffixes="eth1!-eth1" nictypes="eth1!ethernet"
These values will be applied to each node in the "compute" group. So, for example, if I list the attribute of "compute08" I'd see the following nic* attribute values set.
lsdef compute08
Object name: compute08
. . . .
nichostnamesuffixes=eth1!-eth1
nicips=eth1!80.0.0.8
nictypes=eth1!ethernet
. . . .
NOTE: Make sure you haven't already set nic* values in the individual node definitions since they would take precedence over the group value.
Limited support for user application network interfaces (work in progress)
TBD
Note: This support is available starting with the xCAT 2.8 release.
????? - can add xCAT network defs for non-management networks - will add domain to resolv.conf files - will add to dhcp domain-search attr
- config provided by user customization scripts
