xCAT Wiki

An extreme cluster/cloud administration toolkit

Brought to you by: besawn, cxhong, gurevich, obihoernchen, victorhu

Ubuntu_Quick_Start

There is a newer version of this page. You can find it here.

xCAT installation using Ubuntu
Prepare for xCAT installation
Install xCAT
Configure xCAT
Deploying Nodes
**Installing Stateful Nodes **
**Installing Stateless Nodes **
Installing other packages with Ubuntu official mirror

xCAT installation using Ubuntu

This is a copy of the iDataplex Quick Start but with Ubuntu and under construction

This document is specific to Ubuntu deployment in statefull installation.

This configuration will have a single dx360 Management Node with 2 other dx360 servers as nodes. The OS deployed will be Ubuntu 12.04.1 LTS (Precise), amd64 Server edition.

Prepare for xCAT installation

xCAT install process will scan and populate certain settings from the running configuration. Having the networks configured ahead of time will aid in correct configuration. All the networks in the cluster must be defined in the xCAT networks table before start installing cluster nodes. When xCAT is installed on the Management Node, it will run makenetworks to create an entry in the networks table for each of the networks the management node is on. Additional network configurations can be added to the xCAT networks table manually later if needed.

Install the Management Node OS

Install a ubuntu as the Management Node (MN). If the Management Node can not access the network, ensure that isc-dhcp-server, apache2, nfs-kernel-server, nmap, bind9,, expect, apache2, xinetd, tftpd-hpa, tftp-hpa, libnet-telnet-perl, syslinux, libsys-virt-perl and perl-XML-Parser are installed. It is recommended the Management Node can access the internet and the external repository, these installs will happen automatically later if not done now. It is recommened to add all mirror source about the current distribution in to the source.list file(such like 'precise main', 'precise-updates main', 'precise universe', 'precise-updates universe'). Refer the Official Archive Mirrors page to find the correct repository. For example:

    deb http://us.archive.ubuntu.com/ubuntu/ precise main
    deb http://us.archive.ubuntu.com/ubuntu/ precise-updates main 
    deb http://us.archive.ubuntu.com/ubuntu/ precise universe
    deb http://us.archive.ubuntu.com/ubuntu/ precise-updates universe

Configure /bin/sh

After Ubuntu6.10, /bin/sh default links to /bin/dash (More information about dash refer here please). All xCAT's scripts were developed under bash, so change the link to /bin/bash. 2 methods:

    cd /bin/ && ln -fs bash sh

    dpkg-reconfigure dash

and select [no]

Configure NICS

Configure the cluster facing nics, the interface which used for managing the compute node should use the static IP address. An example /etc/network/interfaces:

    auto eth1
    iface eth1 inet static
      address 10.168.52.1
      netmask 255.255.255.0

Configure hostname

The xCAT 2 Management node hostname must be configured before setting up the xCAT 2 Management Node. The hostname or its resolvable ip address will be used as the default master name in the xCAT site table, when installed. This name needs to be the one that will resolve to the cluster-facing NIC. Short hostnames ( no domain) are the norm for the Management Node and all cluster nodes.

/etc/hostname should have as an example

mgt

If you run hostname command, if should return the same:

    hostname
    mgt

Configure DNS resolution

In the /etc/resolv.conf file, set the nameserver to the management node. For example:

    search cluster
    nameserver 10.168.52.1

Configure NTP

When using management node to install service nodes or compute nodes, the timezone configuration on the management node will be inherited by the service nodes or compute nodes. So it is recommended to configure time management on the management node.

Setup basic hosts file

Ensure lines like the following is in /etc/hosts:

    127.0.0.1               localhost.localdomain localhost
    ::1                     localhost6.localdomain6 localhost6
    ###
    10.168.52.1 mgt.cluster mgt

Setup the TimeZone

    sudo dpkg-reconfigure tzdata

Create a Separate File system for /install (optional)

It is not required, but recommended, that you create a separate file system for the /install directory on the Management Node. The size should be at least 30 meg to hold to allow space for several install images.

Install xCAT

Option 1: Prepare to Install xCAT Directly from the Internet-Hosted Repository

Add the following line into /etc/apt/sources.list.d/xcat-core.list:

    deb [arch=amd64] http://jaist.dl.sourceforge.net/project/xcat/ubuntu/devel/core-snap precise main

Add the following line into /etc/apt/sources.list.d/xcat-dep.list:

    deb [arch=amd64] http://jaist.dl.sourceforge.net/project/xcat/ubuntu/xcat-dep precise main

Option 2: Download the xCAT Software

If not able to, or not wishing to, use the live internet repository, choose this option.

Go to the [Download_xCAT] site and download the level of xCAT ubuntu tarball you desire. Go to the xCAT Dependencies Download page and download the latest snap of the xCAT ubuntu dependency tarball. (The latest snap of the xCAT dependency tarball will work with any version of xCAT.)

Copy the files to the Management Node (MN) and untar them:

    mkdir /root/xcat2
    cd /root/xcat2
    tar jxvf xcat-core-ubuntu*.tar.bz2     # or core-debs-snap.tar.bz2
    tar jxvf xcat-dep-ubuntu-*.tar.bz2

Setup Local Repositories for xCAT and Dependencies

Point apt to the local repositories for xCAT and its dependencies:

    cd /root/xcat2/xcat-core
    ./mklocalrepo.sh
    cd /root/xcat2/xcat-dep
    ./mklocalrepo.sh

Configure the apt key

Add the gpg public key for apt to load xCAT's packages information. Or when run the 'apt-get update' will display some warning messages (The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 20E475A8DA736C68).

    wget -O - "http://sourceforge.net/projects/xcat/files/ubuntu/apt.key/download" | apt-key add -

Install xCAT packages

Use apt to install xCAT and all the dependencies for you:

    apt-get clean all
    apt-get update
    apt-get --allow-unauthenticated install xcat

This should automatically grab all the relevant dependencies from the web if required.

Test xCAT Installation

Add xCAT commands to the path by running the following:

    source /etc/profile.d/xcat.sh

Check to see the database is initialized:

    tabdump site

The output should similar to the following:

    key,value,comments,disable
    "xcatdport","3001",,
    "xcatiport","3002",,
    "tftpdir","/tftpboot",,
    "installdir","/install",,
         .
         .
         .

Updating xCAT Packages Later

If you need to update the xCAT software later:

    apt-get update
    apt-get --allow-unauthenticated install xcat perl-xcat xcat-server xcat-client \
       xcat-genesis-scripts-amd64 xcat-genesis-base-amd64

Configure xCAT

Networks Table

All networks in the cluster must be defined in the networks table. When xCAT was installed, it ran makenetworks, which created an entry in this table for each of the networks the management node is on. Now is the time to add or update any other networks needed to the networks table.

For a sample Networks Setup, see the following example:
Setting_Up_a_Linux_xCAT_Mgmt_Node#Appendix_A:_Network_Table_Setup_Example

passwd Table

This includes the system entry which is the passwd that will be assigned to root when the node is installed. You can modify this table, to change the default password for root.

    tabedit passwd
    key,username,password,cryptmethod,comments,disable
    "system","root","cluster",,,
    "ipmi","USERID","PASSW0RD",,,

Setup DHCP

This will get the network stanza part of the DHCP configuration (including the dynamic range) set:

    makedhcp -n

Setup TFTP

Nothing to do here - the TFTP server is done by xCAT during the Management Node install.

Deploying Nodes

Add nodes to nodelist

Here you can use the power of the templates to define the nodes quickly:

    mkdef -t node -o n1-n10 groups=compute,all

At this point, xCAT should be ready to begin managing services.

To see your nodelist

    nodels

    lsdef all

Setup /etc/hosts file

Add the map between the xCAT node names and IP addresses,

    chdef -t node -o n1 ip=xxx.xxx.xxx.xxx
    chdef -t node -o n2 ip=xxx.xxx.xxx.xxx
                   .
                   .
                   .

Verify the entries have been created in the file /etc/hosts. For example your /etc/hosts should look like this:

    127.0.0.1               localhost.localdomain localhost
    ::1                     localhost6.localdomain6 localhost6
    ###
    10.168.52.1 mgt mgt.cluster
    10.168.52.101 n1 n1.cluster
    10.168.52.102 n2 n2.cluster
    10.168.52.103 n3 n3.cluster
                  .
                  .
                  .

Setup DNS

To get the hostname/IP pairs copied from /etc/hosts to the DNS on the MN:

Ensure that /etc/bind/named does not have ROOTDIR set
Set site.forwarders to your site-wide DNS servers that can resolve site or public hostnames. The DNS on the MN will forward any requests it can't answer to these servers.

    chdef -t site forwarders=1.2.3.4,1.2.5.6

Run makedns

    makedns -n

Setup DHCP

This mac address can be obtained from the back panel of the machine. This MAC address should belong to NIC which is connected to the management network.

    chdef n1 mac="xx:xx:xx:xx:xx:xx"
    chdef n2 mac="yy:yy:yy:yy:yy:yy"
      .
      .
      .



  * Add the nodes to dhcp 

    makedhcp compute

Map between nodes and bmcs

    chdef n1 bmc=xxx.xxx.xxx.xxx mgt=ipmi
    chdef n2 bmc=xxx.xxx.xxx.xxx mgt=ipmi
      .
      .
      .

Declare use of SOL

If not using a terminal server, SOL is recommended, but not required to be configured. To instruct xCAT to configure SOL in installed operating systems on dx340 systems:

    chdef -t group -o compute serialport=1 serialspeed=19200 serialflow=hard

noderes Table

A basic noderes table which defines node resources during install. If the server is not defined, it will default to the Management Node.

Create Ubuntu repository

Download Ubuntu ISOs and place in a directory:

    mkdir /root/xcat2
    cd /root/xcat2 
    wget <ISO of your Ubuntu OS>

Run copycds to setup the install directory for the node diskfull/diskless boots. The copycds commands will copy the contents of to /install/ubuntu12.04.1/x86_64/. For example:

    cd /root/xcat2
    copycds ubuntu-12.04.1-server-amd64.iso

Installing Stateful Nodes

Begin Installation

Run "lsdef -t osimage" to list all osimage object created by command copycds.

    lsdef -t osimage
    ubuntu12.04.1-x86_64-install-compute  (osimage)
    ubuntu12.04.1-x86_64-netboot-compute  (osimage)
    ubuntu12.04.1-x86_64-statelite-compute  (osimage)

Set up pkglists

You likely want to customize the main pkglist for the image. Copy the default pkglist file "/opt/xcat/share/xcat/install/ubuntu/compute.pkglist" to a directory, and edit the new pkglist file(the default packages' name can not be deleted). This is the list of debs that will be installed from the distro. (Other debs that they depend on will be installed automatically.) For example:

    mkdir -p /install/custom/install/ubuntu12.04.1/
    cp -p /opt/xcat/share/xcat/install/ubuntu/compute.pkglist /install/custom/install/ubuntu12.04.1/
    vi /install/custom/install/ubuntu12.04.1/compute.pkglist
    chdef -t osimage <osimage name> pkglist=/install/custom/install/ubuntu12.04.1/compute.pkglist

Installing Additional Packages Using an Otherpkgs Pkglist

If you have additional rpms (rpms not in the distro) that you also want installed, make a directory to hold them, create a list of the packages you want installed, and add that information to the osimage definition:

Create a directory to save all packages which will be installed(the directory should under /install/).

    /install/post/otherpkgs/ubuntu/x86_64

Download all deb packages from the internet(all dependent packages should be downloaded also).

    apt-rdepends <package name< | grep -v Depends #find out all dependencies
    apt-get download -d <all packages' name>  #download all needed packages

Install dpkg-dev on the MN.

    apt-get install dpkg-dev

Go to the directory and run 'dpkg-scanpackages' to create the repository for apt.

    cd /install/post/otherpkgs/ubuntu/x86_64
    dpkg-scanpackages . > Packages  (this command can not be changed, Packages is a key word for apt)

Create an other packages list file and add the pakcages' name to this list file.

vi /install/custom/install/ubuntu/compute.otherpkgs.pkglist
Use chdef command to define the otherpkgdir and otherpkglist for the osimage object.

    chdef -t osimage <osimage name> otherpkgdir="/install/post/otherpkgs/ubuntu/x86_64" \
     otherpkglist="/install/custom/install/ubuntu/compute.otherpkgs.pkglist"

If you already have a different OS on your nodes and you haven't configured your nodes to always boot from the network, then run rsetboot to instruct them to boot from the network for the next boot:

    rsetboot compute net

The nodeset command tells xCAT what you want to do next with this node, and powering on the node starts the installation process:

    nodeset compute osimage=<osimage name>
    rpower compute boot

Tip: when nodeset is run, it processes the preseed template associated with the osimage, plugging in node-specific attributes, and creates a specific preseed file for each node in /install/autoinst. If you need to customize the template, make a copy of the file that is pointed to by the osimage.template and edit that file (or the files it includes).

Installing other packages from the internet repository, refer Ubuntu_Quick_Start/#installing-other-packages-with-ubuntu-official-mirror.

Monitor installation

rcons to watch one node

    rcons n1

Installing Stateless Nodes

list all images

Default osimage objects are also defined when copycds is run. To view the osimages:

    lsdef -t osimage          # see the list of osimages
    lsdef -t osimage <osimage-name>          # see the attributes of a particular osimage

If the image you are building is for nodes that are the same architecture as the management node (the most common case), then you can follow the instructions below to run genimage on the management node. If you are building an image for a different architecture, [Building_a_Stateless_Image_of_a_Different_Architecture].

Set up pkglists

You likely want to customize the main pkglist for the image. This is the list of rpms or groups that will be installed from the distro. (Other rpms that they depend on will be installed automatically.) For example:

    mkdir -p /install/custom/netboot/ubuntu/
    cp -p /opt/xcat/share/xcat/netboot/ubuntu/compute.ubuntu12.04.1.pkglist /install/custom/netboot/ubuntu/
    vi /install/custom/netboot/ubuntu/compute.pkglist
    chdef -t osimage <osimage name> pkglist=/install/custom/netboot/ubuntu/compute.pkglist

Installing Additional Packages Using an Otherpkgs Pkglist

If you have additional rpms (rpms not in the distro) that you also want installed, make a directory to hold them, create a list of the rpms you want installed, and add that information to the osimage definition:

Create a directory to save all packages which will be installed(the directory should under /install/).

    /install/post/otherpkgs/ubuntu/x86_64

Download all deb packages from the internet(all dependent packages should be downloaded also).
Install dpkg-dev on the MN.

    apt-get install dpkg-dev

Go to the directory and run 'dpkg-scanpackages' to create the repository for apt.

    cd /install/post/otherpkgs/ubuntu/x86_64
    dpkg-scanpackages . > Packages  (this command can not be changed, Packages is a key word for apt)

Generate and pack your image

Run genimage to generate the image based on the mycomputeimage definition:

    genimage ubuntu12.04.1-x86_64-netboot-compute

Before you pack the image, you have the opportunity to change any files in the image that you want to, by cd'ing to the rootimgdir (e.g. /install/netboot/rhels6/x86_64/compute/rootimg).Although, instead, we recommend that you make all changes to the image via your postinstall script, so that it is repeatable.

The genimage command creates /etc/fstab in the image. If you want to, for example, limit the amount of space that can be used in /tmp and /var/tmp, you can add lines like the following to it (either by editing it by hand or via the postinstall script):

    tmpfs   /tmp     tmpfs    defaults,size=50m             0 2
    tmpfs   /var/tmp     tmpfs    defaults,size=50m       0 2

But probably an easier way to accomplish this is to create a postscript to be run when the node boots up with the following lines:

    logger -t xcat "$0: BEGIN"
    mount -o remount,size=50m /tmp/
    mount -o remount,size=50m /var/tmp/
    logger -t xcat "$0: END"

Assuming you call this postscript settmpsize, you can add this to the list of postscripts that should be run for your compute nodes by:

    chdef -t group compute -p postscripts=settmpsize

Now pack the image to create the ramdisk:

    packimage ubuntu12.04.1-x86_64-netboot-compute

Boot the nodes

    nodeset compute osimage=ubuntu12.04.1-x86_64-netboot-compute
    rsetboot compute net
    rpower compute boot

Update your diskless image later

If you need to update your diskless image sometime later, change your osimage attributes and the files they point to accordingly, and then rerun genimage, packimage, nodeset, and rpower.

Installing other packages with Ubuntu official mirror

Specify the repository

Use the internet repository directly when define the otherpkgdir attribute:

    chdef -t osimage <osimage name> otherpkgdir="http://us.archive.ubuntu.com/ubuntu/ precise main,http://us.archive.ubuntu.com/ubuntu/ precise-update main"

Specify otherpkglist file

create an otherpkglist file, /install/custom/install/ubuntu/compute.otherpkgs.pkglist. Add the packages' name into thist file. And modify the otherpkglist attribute for osimage object.

    chdef -t osimage <osimage name> otherpkglist=/install/custom/install/ubuntu/compute.otherpkgs.pkglist

Compute nodes can not access the internet

Use apt proxy

Step 1: Install Squid on the server which can access the internet (Here uses management node as the proxy server)

    apt-get install squid

Step 2: Edit the Squid configuration file /etc/squid3/squid.conf, find the line "#http_access deny to_localhost". Add the following 2 lines behind this line.

    acl cn_apt src <compute node sub network>/<net mask length>
    http_access allow cn_apt

For more refer Squid configuring.

Step 3: Restart the proxy service

    service squid3 restart

Step 4: Create a postscript under /install/postscripts/ directory, called aptproxy, add following lines

    #!/bin/sh
    PROXYSERVER=$1
    if [ -z $PROXYSERVER ];then
        PROXYSERVER=$MASTER
    fi

    PROXYPORT=$2
    if [ -z $PROXYPORT ];then
        PROXYPORT=3128
    fi

    if [ -e "/etc/apt/apt.conf" ];then
        sed '/^Acquire::http::Proxy/d' /etc/apt/apt.conf &gt; /etc/apt/apt.conf.new
        mv -f /etc/apt/apt.conf.new /etc/apt/apt.conf
    fi
    echo "Acquire::http::Proxy \"http://${PROXYSERVER}:$PROXYPORT\";" &gt;&gt; /etc/apt/apt.conf

Step 5: add this postscript to compute nodes, the [proxy server ip] and [proxy server port] are optional parameters for this postscript. If they are not specified, xCAT will use the management node ip and 3128 by default.

    chdef <node range> -p postscripts="aptproxy [proxy server ip] [proxy server port]"

Step 6: Edit the otherpkglist file, add the require software packages' name.

Step 7: Edit the otherpkgdir attribute for os image object, can use the internet repositories directly.

Step 8: Run nodeset, rsetboot, rpower commands to provision the compute nodes.

Use local mirror

Find a server witch can connect the internet, and can be accessed by the compute nodes.

step 1: Install apt-mirror

    apt-get install apt-mirror

step 2: Configure apt-mirror

    vim /etc/apt/mirror.list

step 3: Run apt-mirror to download the repositories(The needed space can be found in Ubuntu Mirrors )

    apt-mirror /etc/apt/mirror.list

step 4: Install apache

    apt-get install apache2

step 5: Setup links to link our local repository folder to our shared apache directory

    ln -s /var/spool/apt-mirror/mirror/archive.ubuntu.com /var/www/archive-ubuntu

When setting the otherpkgdir attribute for the osimages, can use http://<local mirror server ip>/archive-ubuntu/ precise main

For more information about setting local repository mirror can refer How to Setup Local Repository Mirror

xCAT Wiki

An extreme cluster/cloud administration toolkit

Ubuntu_Quick_Start

xCAT installation using Ubuntu

Prepare for xCAT installation

Install the Management Node OS

Configure /bin/sh

Configure NICS

Configure hostname

Configure DNS resolution

Configure NTP

Setup basic hosts file

Setup the TimeZone

Create a Separate File system for /install (optional)

Install xCAT

Option 1: Prepare to Install xCAT Directly from the Internet-Hosted Repository

Option 2: Download the xCAT Software

Setup Local Repositories for xCAT and Dependencies

Configure the apt key

Install xCAT packages

Test xCAT Installation

Updating xCAT Packages Later

Configure xCAT

Networks Table

passwd Table

Setup DHCP

Setup TFTP

Deploying Nodes

Add nodes to nodelist

Setup /etc/hosts file

Setup DNS

Setup DHCP

Map between nodes and bmcs

Declare use of SOL

noderes Table

Create Ubuntu repository

**Installing Stateful Nodes **

Begin Installation

Set up pkglists

Installing Additional Packages Using an Otherpkgs Pkglist

Monitor installation

**Installing Stateless Nodes **

list all images

Set up pkglists

Installing Additional Packages Using an Otherpkgs Pkglist

Generate and pack your image

Boot the nodes

Update your diskless image later

Installing other packages with Ubuntu official mirror

Specify the repository

Specify otherpkglist file

Compute nodes can not access the internet

Installing Stateful Nodes

Installing Stateless Nodes