From xcat

Jump to: navigation, search

The Kernel-based Virtual Machine (KVM) is the way forward for virtualization technology for Enterprise Linux distributions For this reason we recommend KVM over other open source technologies like Xen (Although there is support for Xen in xCAT)
KVM is included in RedHat 5 as well as SLES 11. KVM is in the Linux kernel as of 2.6.20 . The xCAT based KVM solution offers users the ability to:
- provision bare metal hypervisors
- provision virtual machines
- migrate virtual machines to different hosts
- install all versions of Linux supported in the standard xCAT provisioning methods (you can install stateless virtual machines, iSCSI, and even regular virtual machines)
- copy on write instances of virtual machines
- copy virtual machines
For best use with xCAT our architecture has the following components:
1. A shared filesystem for hosting virtual machines. This should be the same directory for all nodes. Having this shared file system (which we recommend be on a SAN, NAS or GPFS allows us migrate nodes from ).
2. xCAT gPXE boot and KVM that comes packaged in the xCAT-deps distribution. This has updated KVM support that plugs nicely into xCAT
3. RedHat or CentOS 5.3 or greater.

Setup the Management Server for KVM

To run KVM you will need a newer Intel or AMD processor with virtualization technology. (Intel VT or AMD-V).
The easiest way to get up and running is to use the /install directory and share it out via NFS to the other hypervisors in the node farm. That is what we'll do for this example.

mkdir /install/vms

Next, export this directory:

echo "/install/vms *(rw,no_root_squash,sync)" >>/etc/exports
exportfs -r

You'll also need to install a few packages that are required for KVM:

yum -y install iscsi-initiator-utils bridge-utils kvm perl-Sys-Virt

Make sure that when you install these packages that the kvm is taken from the xCAT dependencies directory instead of from the operating system directory. Make sure that the kvm RPM is equal or greater than kvm-85-1

Install the Hypervisor

We create a hypervisor template and install this on the node.

mkdir -p /install/custom/centos/
cp /opt/xcat/share/xcat/install/rh/kvm.tmpl
cp -a /opt/xcat/share/xcat/install/scripts/ \
/install/custom/install/

We will now modify this kvm.tmpl to install the hypervisor.

1. Add the xCAT-dep repo

Since this is CentOS5, our xCAT-dep repo is /install/xcat/xcat-dep/rh5/x86_64/. This is local on the machine and as such is in a directory that happens to be in the apache directory, so I can serve out this directory with http. As such, I add the line:

repo --name=xcat-dep \
--baseurl=http://#TABLE:noderes:$NODE:nfsserver#/install/xcat/xcat-dep/rh5/#TABLE:nodetype:$NODE:arch#

After the initial --url directory.

2. Add kvm packages

Now that we have our additional repos, we add a few more packages to this kickstart template:

bridge-utils
dnsmasq
iscsi-initiator-utils
kvm
perl-Sys-Virt
libvirt.x86_64
gpxe-kvm

These are the packages required for kvm.

3. Modify the post install scripts

xCAT comes with a post install script called mkhyperv which does some initial setup (this script was checked into SVN on 5/7/09, so check there if you don't have it). You should be able to use it as is, so long as the node's primary interface is eth0. You may want to look over it and change it, especially if you are planning on doing some funky networking.
Once you dig it, add it to your post installation scripts:

nodech x01 postscripts.postscripts=mkhyperv

4. Export shared directory (recommended, but you can work around it if you create the images your self)

To run the xCAT commands to create virtual machines, the nodes and the management server require a shared filesystem viewable in the same directory. An easy way to do this is to create another post install script to mount this directory. We made one and called it mountvms:

#!/bin/sh
logger -t xcat "Install: setting vms mount in fstab"
mkdir -p /install/vms
echo "$MASTER:/install/vms /install/vms nfs rsize=8192,wsize=8192,timeo=14,intr,nfsvers=2 1 2" >>/etc/fstab

The above script just creates a directory called /install/vms and then mounts this from the management server. If you have a filer or another storage device where you want all your virtual machines to go, then you can change the scripts according to your needs.

# chmod 755 mountvms
# nodech x01 postscripts.postscripts,=mountvms

5. Install the node

# rinstall x01 -o centos5.3 -a x86_64 -p kvm

The kvm.tmpl will be pulled from the /install/custom/install/centos directory and the node will provision. When finished with the node you can ssh to it an verify it was setup correctly by running:

# virsh list

You should see some nice output that signals that its up and running.
If you see something different then something went wrong!

You should also notice that when you run the ifconfig command you'll have a br0 interface:

# ifconfig
br0 Link encap:Ethernet HWaddr 00:14:5E:55:5B:AC
inet addr:192.168.15.72 Bcast:192.168.15.255 Mask:255.255.255.0
inet6 addr: fe80::214:5eff:fe55:5bac/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:1930 errors:0 dropped:0 overruns:0 frame:0
TX packets:846 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:166778 (162.8 KiB) TX bytes:95512 (93.2 KiB)

eth0 Link encap:Ethernet HWaddr 00:14:5E:55:5B:AC
inet6 addr: fe80::214:5eff:fe55:5bac/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:5072 errors:0 dropped:0 overruns:0 frame:0
TX packets:1160 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:412342 (402.6 KiB) TX bytes:132939 (129.8 KiB)
Interrupt:66

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:56 errors:0 dropped:0 overruns:0 frame:0
TX packets:56 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:6151 (6.0 KiB) TX bytes:6151 (6.0 KiB)

If you don't have that then you probably didn't use the xCAT post install script. You can hack it together quickly by running:

IPADDR=172.20.1.19/16
brctl addif vlan1 eth0
brctl addbr vlan1
brctl setfd vlan1 0
ip addr add dev vlan1 $IPADDR
brctl addif vlan1 eth0
ip link set vlan1 up
ip addr del dev eth0 $IPADDR

Create and Install Virtual Machines

Now the fun begins as we install virtual machines on our hypervisor(s).

1. Define virtual machines to the system

For my test case, I'll create 4 of them. First I add them to /etc/hosts:

tail -4 /etc/hosts
192.168.15.81 v1
192.168.15.82 v2
192.168.15.83 v3
192.168.15.84 v4

Now add them to xCAT under the vm group:

nodeadd v1-v4 groups=vm

Next, update DNS with this new node:

makedns
service named restart

2. Define virtual machine in xCAT: The vm table

We start with the vm table. Ours looks like this:

tabedit vm
#node,host,migrationdest,storage,memory,cpus,nics,bootorder,clockoffset,virtflags,vncport,textconsole,beacon,comments,disable
v1,x02,x01-x02,/install/vms/v1,1024,2,br0,"network,hd"
v2,x02,x01-x02,/install/vms/v2,1024,2,br0,"network,hd"
v3,x02,x01-x02,/install/vms/v3,1024,2,br0,"network,hd"
v4,x02,x01-x02,/install/vms/v4,1024,2,br0,"network,hd"

The table entries are defined as follows:
node - this is the virtual machine's name. We have v1-v2.
host - this is the hypervisor. In our case we'll install these machines on the node x01
migrationdest - this is an xCAT node range of which nodes this virtual machine is allowed to run on. We have two machines: x01 and x02.
storage - this is the file that holds all the virtual machine information.
memory - this is how much memory the virtual machine gets in Megabytes. We're giving each machine here 1GB.
cpus - the number of CPU's per machine
nics - this should match the NIC that you defined on the hypervisor. If you used the post installation script mkhyperv then this is br0
bootorder - you can set the boot order of the machines! To install, you may want to have them network boot. When they network boot, they boot just like real machines.
The other flags can be left alone.

3. Define virtual machine in xCAT: The nodehm table

Virtual machines done with KVM require the following entry in the nodehm table:

"vm",,"kvm",,,,,"0","115200",,,,

Since all our virtual machines belong to the vm group, then with one line, we defined their management method and the way we'll talk to them remotely. For all virtual machines in KVM, the baud rate should be 115200.

4. Define the virtual machine in xCAT: The nodetype table

Let's just do the standard compute image: Our nodetype table has this entry for our VM machines:

vm,centos5.3,x86_64,compute

5. Make the virtual machine, boot it up, and watch it go!

mkvm v1 -s 20G
rpower v1 boot
wvid v1

As it boots up, you should see a screen pop up!

Next, you should be able to look at the node in rcons/wcons

mkconservercf v1-v2
wcons v1

Now, have a look as it boots up, you'll see it got the xCAT standby kernel!

6. Installation

Well, now we're just back to the old game of installing. One thing to notice is that automated discovery is built in. You can look and see that you already have a MAC address for this machine! (tabdump mac | grep v1). It even set up dhcp for you!
So, from here you can just install the node!

rinstall v1

Then the node will automatically reboot and install. You'll have a normal node!

7. Possible Errors

One error we saw on occasion was the following message:

# rpower v1 on
v1: internal error no supported architecture for os type 'hvm'

This error was fixed by restarting libvirtd on the host machine

# ssh x01 service libvirtd restart

KVM