Recent changes to InstallServer

InstallServer modified by Lissa Valletta

Lissa Valletta — Thu, 04 Sep 2014 15:34:50 -0000

--- v2
+++ v3
@@ -2,21 +2,26 @@

 ## Intro

-Installing an Install Server makes your cluster scale. In this example I have 2 install servers. Each install server is an install server for 128 nodes. This for SLES10.1. If you're running another OS, mkinstall probably won't work for you. Use the default in /opt/xcat/share/xcat/netboot/rh. 
-
-Also note, that you'll need genimage.yum. If you use something else, then change the below script. This should be used as a reference and not copied directly. 
+Installing an Install Server makes your cluster scale. In this example I have 2 install servers. Each install server is an install server for 128 nodes. This for SLES10.1. If you're running another OS, mkinstall probably won't work for you. Use the default in /opt/xcat/share/xcat/netboot/rh.
+
+Also note, that you'll need genimage.yum. If you use something else, then change the below script. This should be used as a reference and not copied directly.

 ## Make A diskless Install Server Image

-Here is the code I used to make a diskless install image. To run it just run:  
-./mkinstall <new image name>
-
-This is not tested anywhere but on my cluster. YMMV. 
+Here is the code I used to make a diskless install image. To run it just run:
+
+~~~~
+./mkinstall <new image="" name="">
+
+~~~~
+
+This is not tested anywhere but on my cluster. YMMV.

 **mkinstall**
-    
+
+~~~~
     #!/bin/bash
-    
+
     NAME=$1
     GENIMAGEDIR=/opt/xcat/share/xcat/netboot/sles
     GENIMAGE=$GENIMAGEDIR/genimage.yum
@@ -26,12 +31,12 @@
     sleep 5
     #MASTERIP=10.4.2.30
     ERROR=0
-    
-    
-    if [ ! -f $GENIMAGEDIR/$NAME.pkglist ]
+
+
+    if [ ! -f $GENIMAGEDIR/$NAME.pkglist ]
     then
      echo "Creating $GENIMAGEDIR/$NAME.pkglist"
-     cat > "$GENIMAGEDIR/$NAME.pkglist" <EOF
+     cat > "$GENIMAGEDIR/$NAME.pkglist" <EOF atftp="" bash="" bind="" @@="" -93,11="" +98,11="" @@="" dhcp-server="" EOF="" fi="" -="" -="" if="" [ !="" -f="" $GENIMAGEDIR="" $NAME.exlist="" ]="" +="" +="" if="" [="" !="" -f="" $GENIMAGEDIR="" $NAME.exlist="" ]="" then="" echo="" "Creating="" $GENIMAGEDIR="" $NAME.exlist"="" -="" cat="" &gt;="" "$GENIMAGEDIR="" $NAME.exlist"="" &lt;EOF2="" +="" cat=""> "$GENIMAGEDIR/$NAME.exlist" <EOF2 .="" usr="" share="" man*="" .="" usr="" share="" locale*="" .="" usr="" share="" i18n*="" @@="" -111,118="" +116,128="" @@="" .="" usr="" lib="" locale*="" .="" boot*="" EOF2="" -="" +="" fi="" -="" -="" +="" +="" $GENIMAGE="" -i="" eth0="" -n="" tg3,bnx2="" -o="" sles10.1="" -p="" $NAME="" chroot="" $IMGROOT="" insserv="" boot.localnet="" chroot="" $IMGROOT="" insserv="" haldaemon="" chroot="" $IMGROOT="" insserv="" dbus="" chroot="" $IMGROOT="" insserv="" network="" -="" +="" #="" change="" syslog="" -="" echo="" "*.*="" @$MASTERIP"="" &gt;="" $IMGROOT="" etc="" syslog.conf="" +="" echo="" "*.*="" @$MASTERIP"=""> $IMGROOT/etc/syslog.conf
     chroot $IMGROOT insserv syslog
-    
-    
+
+
     chroot $IMGROOT insserv portmap
     chroot $IMGROOT insserv sshd
-    
+
     # NTP
-    echo "server $MASTERIP" >> $IMGROOT/etc/ntp.conf
+    echo "server $MASTERIP" >> $IMGROOT/etc/ntp.conf
     cp /etc/localtime $IMGROOT/etc/
     cp /etc/hosts $IMGROOT/etc/
     chroot $IMGROOT insserv ntp
     chroot $IMGROOT insserv apache2
-    
+
     # stop dhcp from starting up until xCAT does it.
     chroot $IMGROOT chkconfig dhcpd off
     chroot $IMGROOT chkconfig dhcrelay off
-    
+
     # copy head nodes sysctl for kernel params
     cp /etc/sysctl.conf $IMGROOT/etc/
-    
+
     # add more nfs threads
     perl -pi -e 's/USE_KERNEL_NFSD_NUMBER="4"/USE_KERNEL_NFSD_NUMBER="64"/g' $IMGROOT/etc/sysconfig/nfs
-    
+
     # dhcp interface assignment
     perl -pi -e 's/DHCPD_INTERFACE=""/DHCPD_INTERFACE="eth0"/g' $IMGROOT/etc/sysconfig/dhcpd
-    
+
     # NFS vodoo
-    echo '/install *(ro,no_root_squash,sync,fsid=13)' >>$IMGROOT/etc/exports
-    
+    echo '/install *(ro,no_root_squash,sync,fsid=13)' >>$IMGROOT/etc/exports
+
     # FSTAB vodoo
     perl -pi -e 's/tmpfs/#tmpfs/g' $IMGROOT/etc/fstab
-    echo "$NAME   /   tmpfs rw 0 1 " >>$IMGROOT/etc/fstab
-    #echo "none    /tmp    tmpfs   defaults,size=10m 0 2" >>$IMGROOT/etc/fstab
-    #echo "none    /var/tmp    tmpfs   defaults,size=10m 0 2" >>$IMGROOT/etc/fstab
-    
+    echo "$NAME   /   tmpfs rw 0 1 " >>$IMGROOT/etc/fstab
+    #echo "none    /tmp    tmpfs   defaults,size=10m 0 2" >>$IMGROOT/etc/fstab
+    #echo "none    /var/tmp    tmpfs   defaults,size=10m 0 2" >>$IMGROOT/etc/fstab
+
     #HTTP fix
     mv $IMGROOT/etc/httpd/conf.d/xcat.conf $IMGROOT/etc/apache2/conf.d/
-    
-    
+
+
     cp /etc/security/limits.conf $IMGROOT/etc/security
     cp /usr/bin/strace $IMGROOT/usr/bin
     echo "Packing Image..."
     packimage -o sles10.1 -p $NAME -a x86_64
     echo "Install Server Image: $NAME has been created.  Please remember to edit"
-    
+
     #echo "the nodetype table:  e.g.: tabedit nodetype"
     #echo "then run: nodeset service netboot"
     #echo "then reboot service nodes:  rpower, or reboot"
     #echo "make sure tabedit site has installloc set to /install"
     #echo "you should also verify post install scripts."

-  
+~~~~

 ## Edit xCAT Tabs

-I'll show you the important tables here. The rest are just normal. 
+I'll show you the important tables here. The rest are just normal.

 ### postscripts
-    
+
+~~~~
+
     #node,postscripts,comments,disable
     "service","servicenode,xcatclient,xcatserver,setupeth,restartxcat",,

-As of xCAT 2.7, the servicenode postscript calls the xcatclient and xcatserver postscripts, so all three are not needed in the postscript table. The table would look like the following: 
-    
+~~~~
+
+As of xCAT 2.7, the servicenode postscript calls the xcatclient and xcatserver postscripts, so all three are not needed in the postscript table. The table would look like the following:
+
+
+~~~~
     #node,postscripts,comments,disable
     "service","servicenode,setupeth,restartxcat",,

-  
-All the scripts here are included with xCAT. I added setupeth because I wanted to change the GbE cards. Then after I did that I needed to restart xCAT so a made a script to do that. 
-
-Here are the scripts: 
-
-**setupeth**  
-Here I have a 10GbE card that I needed to load the driver on. I also needed to change the way they were ordered because I wanted my 10GbE card to be eth0. So here is how I did it. (Notice this is all in SLES10) 
-    
+~~~~
+
+
+All the scripts here are included with xCAT. I added setupeth because I wanted to change the GbE cards. Then after I did that I needed to restart xCAT so a made a script to do that.
+
+Here are the scripts:
+
+**setupeth**
+Here I have a 10GbE card that I needed to load the driver on. I also needed to change the way they were ordered because I wanted my 10GbE card to be eth0. So here is how I did it. (Notice this is all in SLES10)
+
+
+~~~~
     insmod /xcatpost/myri10ge.ko
     ME=`hostname`
     MMM=`grep $ME-bmm /etc/hosts | awk '{print $1}'`
     TENGE=`grep $ME /etc/hosts | head -1 | awk '{print $1}'`
-    
+
     # flip the udev's around
     cp /etc/udev/rules.d/30-net_persistent_names.rules /tmp/net_persistent_names.rules.ORIG
     perl -pi -e 's/eth0/ethX/g' /etc/udev/rules.d/30-net_persistent_names.rules
     perl -pi -e 's/eth2/eth0/g' /etc/udev/rules.d/30-net_persistent_names.rules
     perl -pi -e 's/eth1/eth2/g' /etc/udev/rules.d/30-net_persistent_names.rules
     perl -pi -e 's/ethX/eth1/g' /etc/udev/rules.d/30-net_persistent_names.rules
-    
+
     echo "BOOTPROTO='static'
     IPADDR=$TENGE
     NETMASK=255.255.0.0
     STARTMODE=auto
     MTU=1500
-    " >/etc/sysconfig/network/ifcfg-eth0
-    
+    " >/etc/sysconfig/network/ifcfg-eth0
+
     echo "BOOTPROTO='static'
     IPADDR=$MMM
     NETMASK=255.255.0.0
     STARTMODE=auto
-    " >/etc/sysconfig/network/ifcfg-eth1
+    " >/etc/sysconfig/network/ifcfg-eth1
     service network stop
     rmmod bnx2
     rmmod myri10ge
@@ -232,45 +247,68 @@
     service network start
     #sleep 10

-**restartxcat**  
-does just what it says it does... oh, and syslog too. 
-    
+~~~~
+
+**restartxcat**
+does just what it says it does... oh, and syslog too.
+
+
+~~~~
     service syslog restart
     service xcatd restart

+~~~~
+
 ### noderes
-    
+
+
+~~~~
     "service",,"pxe",,"10.1.2.30",,,,,,,"eth0","eth0","10.1.2.30",,,,
     "hgroup","dnh01","pxe","dnh01","dnh01",,,,,,,"eth0","eth0","dnh01",,,,
     "igroup","dni01","pxe","dni01","dni01",,,,,,,"eth0","eth0","dni01",,,,

-Here my two install servers are dnh01 and dni01. They service the hgroup and the igroup. Each of these groups has about 128 nodes. The nodes are assigned in the nodelist table. 
+~~~~
+
+Here my two install servers are dnh01 and dni01. They service the hgroup and the igroup. Each of these groups has about 128 nodes. The nodes are assigned in the nodelist table.

 ### servicenode

-Looks like this: 
-    
+Looks like this:
+
+
+~~~~
     #node,nameserver,dhcpserver,tftpserver,nfsserver,conserver,monserver,ldapserver,ntpserver,ftpserver,comments,disable
     "dnh01","0","1","1","1","0","1","0","1","1",,
     "dni01","0","1","1","1","0","1","0","1","1",

+~~~~
+
 ### nodetype
-    
+
+
+~~~~
     #node,os,arch,profile,nodetype,comments,disable
     "service","sles10.1","x86_64","s10","osi",,

-Notice that when I ran mkinstall above I ran it like: 
-    
+~~~~
+
+Notice that when I ran mkinstall above I ran it like:
+
+
+~~~~
     mkinstall s10

-So since that's the image I made, I want to boot to it. 
-
-  
+~~~~
+
+So since that's the image I made, I want to boot to it.
+
+

 ## Installing

-The rest is all just done using normal xCAT commands. Run: nodeset <servicenodes> netboot.  
-The biggest problem I had when doing this initially was that the 10GbE card runs at 9000MTU by default and my switch wasn't set to handle that. So when I added the MTU in there it made everything work fine. 
-
-Hopefully this helps someone else. 
+The rest is all just done using normal xCAT commands. Run: nodeset <servicenodes> netboot.
+The biggest problem I had when doing this initially was that the 10GbE card runs at 9000MTU by default and my switch wasn't set to handle that. So when I added the MTU in there it made everything work fine.
+
+Hopefully this helps someone else.
+

InstallServer modified by Lissa Valletta

Lissa Valletta — Mon, 23 Jun 2014 15:54:49 -0000

--- v1
+++ v2
@@ -186,6 +186,12 @@
     #node,postscripts,comments,disable
     "service","servicenode,xcatclient,xcatserver,setupeth,restartxcat",,

+As of xCAT 2.7, the servicenode postscript calls the xcatclient and xcatserver postscripts, so all three are not needed in the postscript table. The table would look like the following: 
+    
+    #node,postscripts,comments,disable
+    "service","servicenode,setupeth,restartxcat",,
+
+  
 All the scripts here are included with xCAT. I added setupeth because I wanted to change the GbE cards. Then after I did that I needed to restart xCAT so a made a script to do that. 

 Here are the scripts:

InstallServer modified by Bruce

Bruce — Mon, 23 Jun 2014 15:54:48 -0000

Intro
Make A diskless Install Server Image
Edit xCAT Tabs
Installing

Intro

Installing an Install Server makes your cluster scale. In this example I have 2 install servers. Each install server is an install server for 128 nodes. This for SLES10.1. If you're running another OS, mkinstall probably won't work for you. Use the default in /opt/xcat/share/xcat/netboot/rh.

Also note, that you'll need genimage.yum. If you use something else, then change the below script. This should be used as a reference and not copied directly.

Make A diskless Install Server Image

Here is the code I used to make a diskless install image. To run it just run:
./mkinstall <new image name>

This is not tested anywhere but on my cluster. YMMV.

mkinstall

#!/bin/bash

NAME=$1
GENIMAGEDIR=/opt/xcat/share/xcat/netboot/sles
GENIMAGE=$GENIMAGEDIR/genimage.yum
IMGROOT=/install/netboot/sles10.1/x86_64/$NAME/rootimg
MASTERIP=$(grep `hostname` /etc/hosts | head -1 | awk '{print $1}')
echo "master is: $MASTERIP"
sleep 5
#MASTERIP=10.4.2.30
ERROR=0

if [ ! -f $GENIMAGEDIR/$NAME.pkglist ]
then
 echo "Creating $GENIMAGEDIR/$NAME.pkglist"
 cat &gt; "$GENIMAGEDIR/$NAME.pkglist" &lt;EOF
atftp
bash
bind
bind-utils
bzip2
compat-libstdc++
dbus-1
dbus-1-glib
dhclient
dhcp-relay
dhcp
dhcpcd
expat
gcc
gcc-fortran
hal
iputils
kernel
kernel-smp
ksh
libgfortran
libxml2
make
nfs-utils
ntp
numactl
openssh
procps
psmisc
resmgr
rpm
rsh
stunnel
tar
tcsh
tk
vim
cron
vsftpd
wget
perl-DBD-Pg
apache2
xCATsn
conserver
expect
fping
ipmitool
perl-XML-Parser
perl-xCAT
postgresql
postgresql-server
syslinux
xCAT-client
xCAT-nbkernel-x86_64
xCAT-nbroot-core-x86_64
xCAT-nbroot-oss-x86_64
xCAT-server
xCATsn
dhcp-server
EOF
fi

if [ ! -f $GENIMAGEDIR/$NAME.exlist ]
then
  echo "Creating $GENIMAGEDIR/$NAME.exlist"
  cat &gt; "$GENIMAGEDIR/$NAME.exlist" &lt;EOF2
./usr/share/man*
./usr/share/locale*
./usr/share/i18n*
./var/cache/yum*
./usr/share/doc*
./usr/share/gnome*
./usr/share/zoneinfo*
./usr/share/cracklib*
./usr/share/info*
./usr/share/omf*
./usr/lib/locale*
./boot*
EOF2

fi

$GENIMAGE -i eth0 -n tg3,bnx2 -o sles10.1 -p $NAME
chroot $IMGROOT insserv boot.localnet
chroot $IMGROOT insserv haldaemon
chroot $IMGROOT insserv dbus
chroot $IMGROOT insserv network

# change syslog
echo "*.* @$MASTERIP" &gt; $IMGROOT/etc/syslog.conf
chroot $IMGROOT insserv syslog

chroot $IMGROOT insserv portmap
chroot $IMGROOT insserv sshd

# NTP
echo "server $MASTERIP" &gt;&gt; $IMGROOT/etc/ntp.conf
cp /etc/localtime $IMGROOT/etc/
cp /etc/hosts $IMGROOT/etc/
chroot $IMGROOT insserv ntp
chroot $IMGROOT insserv apache2

# stop dhcp from starting up until xCAT does it.
chroot $IMGROOT chkconfig dhcpd off
chroot $IMGROOT chkconfig dhcrelay off

# copy head nodes sysctl for kernel params
cp /etc/sysctl.conf $IMGROOT/etc/

# add more nfs threads
perl -pi -e 's/USE_KERNEL_NFSD_NUMBER="4"/USE_KERNEL_NFSD_NUMBER="64"/g' $IMGROOT/etc/sysconfig/nfs

# dhcp interface assignment
perl -pi -e 's/DHCPD_INTERFACE=""/DHCPD_INTERFACE="eth0"/g' $IMGROOT/etc/sysconfig/dhcpd

# NFS vodoo
echo '/install *(ro,no_root_squash,sync,fsid=13)' &gt;&gt;$IMGROOT/etc/exports

# FSTAB vodoo
perl -pi -e 's/tmpfs/#tmpfs/g' $IMGROOT/etc/fstab
echo "$NAME   /   tmpfs rw 0 1 " &gt;&gt;$IMGROOT/etc/fstab
#echo "none    /tmp    tmpfs   defaults,size=10m 0 2" &gt;&gt;$IMGROOT/etc/fstab
#echo "none    /var/tmp    tmpfs   defaults,size=10m 0 2" &gt;&gt;$IMGROOT/etc/fstab

#HTTP fix
mv $IMGROOT/etc/httpd/conf.d/xcat.conf $IMGROOT/etc/apache2/conf.d/

cp /etc/security/limits.conf $IMGROOT/etc/security
cp /usr/bin/strace $IMGROOT/usr/bin
echo "Packing Image..."
packimage -o sles10.1 -p $NAME -a x86_64
echo "Install Server Image: $NAME has been created.  Please remember to edit"

#echo "the nodetype table:  e.g.: tabedit nodetype"
#echo "then run: nodeset service netboot"
#echo "then reboot service nodes:  rpower, or reboot"
#echo "make sure tabedit site has installloc set to /install"
#echo "you should also verify post install scripts."

Edit xCAT Tabs

I'll show you the important tables here. The rest are just normal.

postscripts

#node,postscripts,comments,disable
"service","servicenode,xcatclient,xcatserver,setupeth,restartxcat",,

All the scripts here are included with xCAT. I added setupeth because I wanted to change the GbE cards. Then after I did that I needed to restart xCAT so a made a script to do that.

Here are the scripts:

setupeth
Here I have a 10GbE card that I needed to load the driver on. I also needed to change the way they were ordered because I wanted my 10GbE card to be eth0. So here is how I did it. (Notice this is all in SLES10)

insmod /xcatpost/myri10ge.ko
ME=`hostname`
MMM=`grep $ME-bmm /etc/hosts | awk '{print $1}'`
TENGE=`grep $ME /etc/hosts | head -1 | awk '{print $1}'`

# flip the udev's around
cp /etc/udev/rules.d/30-net_persistent_names.rules /tmp/net_persistent_names.rules.ORIG
perl -pi -e 's/eth0/ethX/g' /etc/udev/rules.d/30-net_persistent_names.rules
perl -pi -e 's/eth2/eth0/g' /etc/udev/rules.d/30-net_persistent_names.rules
perl -pi -e 's/eth1/eth2/g' /etc/udev/rules.d/30-net_persistent_names.rules
perl -pi -e 's/ethX/eth1/g' /etc/udev/rules.d/30-net_persistent_names.rules

echo "BOOTPROTO='static'
IPADDR=$TENGE
NETMASK=255.255.0.0
STARTMODE=auto
MTU=1500
" &gt;/etc/sysconfig/network/ifcfg-eth0

echo "BOOTPROTO='static'
IPADDR=$MMM
NETMASK=255.255.0.0
STARTMODE=auto
" &gt;/etc/sysconfig/network/ifcfg-eth1
service network stop
rmmod bnx2
rmmod myri10ge
modprobe bnx2
insmod /xcatpost/myri10ge.ko
sleep 5
service network start
#sleep 10

restartxcat
does just what it says it does... oh, and syslog too.

service syslog restart
service xcatd restart

noderes

"service",,"pxe",,"10.1.2.30",,,,,,,"eth0","eth0","10.1.2.30",,,,
"hgroup","dnh01","pxe","dnh01","dnh01",,,,,,,"eth0","eth0","dnh01",,,,
"igroup","dni01","pxe","dni01","dni01",,,,,,,"eth0","eth0","dni01",,,,

Here my two install servers are dnh01 and dni01. They service the hgroup and the igroup. Each of these groups has about 128 nodes. The nodes are assigned in the nodelist table.

servicenode

Looks like this:

#node,nameserver,dhcpserver,tftpserver,nfsserver,conserver,monserver,ldapserver,ntpserver,ftpserver,comments,disable
"dnh01","0","1","1","1","0","1","0","1","1",,
"dni01","0","1","1","1","0","1","0","1","1",

nodetype

#node,os,arch,profile,nodetype,comments,disable
"service","sles10.1","x86_64","s10","osi",,

Notice that when I ran mkinstall above I ran it like:

mkinstall s10

So since that's the image I made, I want to boot to it.

Installing

The rest is all just done using normal xCAT commands. Run: nodeset <servicenodes> netboot.
The biggest problem I had when doing this initially was that the 10GbE card runs at 9000MTU by default and my switch wasn't set to handle that. So when I added the MTU in there it made everything work fine.

Hopefully this helps someone else.