https://sourceforge.net/p/xcat/wiki/Setup_HA_Mgmt_Node_With_Shared_Data/Recent changes to Setup_HA_Mgmt_Node_With_Shared_DataenFri, 08 Aug 2014 13:50:54 -0000https://sourceforge.net/p/xcat/wiki/Setup_HA_Mgmt_Node_With_Shared_Data/<div class="markdown_content"><pre></pre> </div>Lissa VallettaFri, 08 Aug 2014 13:50:54 -0000https://sourceforge.net1b257b7c5d1b781b68261c8e044ecf821ce89968https://sourceforge.net/p/xcat/wiki/Setup_HA_Mgmt_Node_With_Shared_Data/<div class="markdown_content"><pre>--- v31 +++ v32 @@ -1,4 +1,4 @@ -[[img src=Official-xcat-doc.png]] + [TOC] </pre> </div>Lissa VallettaFri, 08 Aug 2014 13:50:27 -0000https://sourceforge.netbf3a067ccc008fdbca02c1f2a8adf21446d401d6https://sourceforge.net/p/xcat/wiki/Setup_HA_Mgmt_Node_With_Shared_Data/<div class="markdown_content"><pre>--- v30 +++ v31 @@ -664,7 +664,7 @@ DNS: run makedns. Verify dns services working for node resolution. Make sure the line "nameserver=&lt;virtual ip&gt;" is in /etc/resolv.conf -For more information on setting up name resolution in an xCAT Cluster: https://sourceforge.net/apps/mediawiki/xcat/index.php?title=Cluster_Name_Resolution +For more information on setting up name resolution in an xCAT Cluster: [Cluster_Name_Resolution] DHCP: if the dhcpsd.cnf(AIX) or dhcpd.leases(Linux) is not syncronized between the primary management node and standby management node, run makedhcp -a to setup the DHCP leases. Verify dhcp is operational. @@ -713,7 +713,7 @@ * [XCAT_Linux_Statelite] -For all the xCAT docs: http://sourceforge.net/apps/mediawiki/xcat/index.php?title=XCAT_Documentation +For all the xCAT docs: [XCAT_Documentation] ## Appendix A Configure Shared Disks </pre> </div>Lissa VallettaTue, 05 Aug 2014 15:35:02 -0000https://sourceforge.net11d4f620d44f587e4e77649a49d0fd75c0136234https://sourceforge.net/p/xcat/wiki/Setup_HA_Mgmt_Node_With_Shared_Data/<div class="markdown_content"><pre>--- v29 +++ v30 @@ -653,6 +653,7 @@ **6\. (DFM only) Setup connection for CEC and Frame** +~~~~ mkhwconn cec,frame -t mkhwconn cec,frame -t -T fnm @@ -675,7 +676,6 @@ [[include ref=HAMN_OS_Image]] -[HAMN_OS_Image] **10\. Restart NFS service and re-export the NFS exports** </pre> </div>Lissa VallettaThu, 24 Jul 2014 12:53:07 -0000https://sourceforge.net2d88eee3f3c86201018d20e42af7b6f70008a0c0https://sourceforge.net/p/xcat/wiki/Setup_HA_Mgmt_Node_With_Shared_Data/<div class="markdown_content"><pre></pre> </div>Lissa VallettaThu, 24 Jul 2014 12:36:27 -0000https://sourceforge.net3e977a82d4b604df5d3313e78ad7bf02b6168009https://sourceforge.net/p/xcat/wiki/Setup_HA_Mgmt_Node_With_Shared_Data/<div class="markdown_content"><pre>--- v27 +++ v28 @@ -674,7 +674,8 @@ This step is required only when you want to use this new primary management node to perform os deployment tasks. [[include ref=HAMN_OS_Image]] - + +[HAMN_OS_Image] **10\. Restart NFS service and re-export the NFS exports** </pre> </div>Lissa VallettaThu, 24 Jul 2014 12:27:32 -0000https://sourceforge.net3a83663cdfabf7b155ba2f7cd72fd13955293fe2https://sourceforge.net/p/xcat/wiki/Setup_HA_Mgmt_Node_With_Shared_Data/<div class="markdown_content"><pre></pre> </div>Lissa VallettaThu, 24 Jul 2014 12:22:02 -0000https://sourceforge.netea30b28b4db0bcf8d17ede865ace8cafc76bae47https://sourceforge.net/p/xcat/wiki/Setup_HA_Mgmt_Node_With_Shared_Data/<div class="markdown_content"><pre>--- v25 +++ v26 @@ -673,27 +673,31 @@ This step is required only when you want to use this new primary management node to perform os deployment tasks. -{{:HAMN_OS_Image}} +[[include ref=HAMN_OS_Image]] + + **10\. Restart NFS service and re-export the NFS exports** Because of the Virtual ip configuration and the other network configuration changes on the new primary management node, the NFS service needs to be restarted and the NFS exports need to be re-exported. On AIX: - + +~~~~ exportfs -ua stopsrc -g nfs startsrc -g nfs exportfs -a - - -On Linux: - +~~~~ + +On Linux: + +~~~~ exportfs -ua service nfs stop service nfs start exportfs -a - +~~~~ ## Setup the Cluster @@ -719,7 +723,8 @@ The operating system is installed on the internal disks. **1\. Connect the first shared drawer and verify you can see the disks.** - + +~~~~ [aixmn1]/# cfgmgr @@ -738,7 +743,7 @@ hdisk11 none None hdisk12 none None hdisk13 none None - +~~~~ **2\. Configure the disks and create the arrays using diag.** @@ -757,6 +762,8 @@ Select the correct sas adapter Select the attached disks. Configuration takes about 40 minutes. + +~~~~ hdisk0 00f604c9b0818d40 rootvg active hdisk1 00f604c9b0819a1d rootvg active @@ -772,23 +779,21 @@ hdisk11 none None hdisk12 none None hdisk13 none None - +~~~~ Create a SAS Disk Array - - sissas1 Available 04-00 PCI Express x8 Ext Dual-x4 3Gb SAS RAID Adapte - - - 6 - +~~~~ + sissas1 Available 04-00 PCI Express x8 Ext Dual-x4 3Gb SAS RAID Adapter + + 6 256 - +~~~~ Select Disks to Use in the Array | - +~~~~ # RAID 6 supports a minimum of 4 and a maximum of 18 disks. pdisk12 00040000 Active Array Candidate 428.4GB Zeroed pdisk13 00040100 Active Array Candidate 428.4GB Zeroed @@ -802,15 +807,16 @@ pdisk21 00040900 Active Array Candidate 428.4GB Zeroed pdisk22 00040A00 Active Array Candidate 428.4GB Zeroed pdisk23 00040B00 Active Array Candidate 428.4GB Zeroed - +~~~~ **3\. Exit and now you should see one disk.** - + +~~~~ [aixmn1]/# lspv hdisk0 00f604c9b0818d40 rootvg active hdisk1 00f604c9b0819a1d rootvg active hdisk2 none None - +~~~~ **4\. Temporarily disconnect the cables to the first drawer. Connect the second drawer and repeat steps. ** @@ -821,22 +827,24 @@ In the following example, hdisk1, hdisk2 and hdisk3 have same PVIDs on both servers. On the primary management node: - + +~~~~ [aixmn1]/# lspv hdisk0 00f604c7c3c22499 rootvg active hdisk1 00f604c7c5b0533a rootvg active hdisk2 00f604c945edde6f None hdisk3 00f604c945ede480 None - +~~~~ On the standby management node: - + +~~~~ [aixmn2]/# lspv hdisk0 00f604c7c3c22499 rootvg active hdisk1 00f604c7c5b0533a rootvg active hdisk2 00f604c945edde6f None hdisk3 00f604c945ede480 None - +~~~~ **6\. Create a volume group and logical volume on the primary management node with no auto varyon during system reboot.** @@ -844,7 +852,7 @@ - +~~~~ [aixmn1]/# mkvg -n -f -y xcatvg hdisk2 hdisk3 0516-1254 mkvg: Changing the PVID in the ODM. @@ -858,17 +866,19 @@ [aixmn1]/# mklv -y loglv00 -t jfs2log -u 1 xcatvg 1 hdisk2 [aixmn1]/# echo y | logform /dev/loglv00 [aixmn1]/# mklv -y xcatlv -t jfs2 -u 2 xcatvg 200 hdisk2 hdisk3 - +~~~~ After the volume group is created, varyon it before creating file systems on it. - + +~~~~ varyonvg xcatvg - +~~~~ **7\. Create xCAT file systems on the primary management node with no auto mount during system reboot.** The following xCAT directories should be put on the shared disk for failover. If you have any applications specific data that need to be shared between the two management nodes, you can put the data on the shared disks also. - + +~~~~ mkdir -p /etc/xcat mkdir -p /install mkdir -p ~/.xcat @@ -881,12 +891,13 @@ mount /install mount /.xcat mount /db2database - +~~~~ **The given sizes are based on setting up a large Power 775 cluster. They may need to be increased or changed during the life of the cluster.** To verify the file systems, check the /etc/filesystems: - + +~~~~ cat /etc/filesystems ... /etc/xcat: @@ -913,39 +924,42 @@ log = /dev/loglv00 mount = false account = false - +~~~~ **8\. Test connectivity from the Standby node. Unmount and Varyoff the volume group on the primary management node.** On the primary management node, enter: - + +~~~~ mount /etc/xcat mount /install mount ~/.xcat mount /db2database varyoffvg xcatvg - +~~~~ **9\. Import the volume group and mount the filesystems on the standby management node. This will import the the volume group and file systems configuration automatically.** On the standby management node, enter: - + +~~~~ importvg -y xcatvg hdisk1 varyonvg xcatvg mount /etc/xcat mount /install mount /.xcat mount /db2database - +~~~~ **10\. Then unmount the file systems and varyoff the xcatvg volume group on the standby management node.** - + + umount /etc/xcat umount /install umount /.xcat umount /db2database varyoffvg xcatvg - +~~~~ ### Configuring Shared Disks on Linux @@ -954,57 +968,63 @@ To verify the shared disks are connected correctly, run the sginfo command on both management nodes and look for the same serial number in the output. Please be aware that the sginfo command may not be installed by default on Linux, the sginfo command is shipped with package sg3_utils, you can manually install the package sg3_utils on both management nodes. Once the sginfo command is installed, run sginfo -l command on both management nodes to list all the known SCSI disks, for example, enter: - + +~~~~ sginfo -l - +~~~~ Output will be similar to: - + +~~~~ /dev/sdd /dev/sdc /dev/sdb /dev/sda /dev/sg0 [=/dev/sda scsi0 ch=0 id=1 lun=0] /dev/sg1 [=/dev/sdb scsi0 ch=0 id=2 lun=0] /dev/sg2 [=/dev/sdc scsi0 ch=0 id=3 lun=0] /dev/sg3 [=/dev/sdd scsi0 ch=0 id=4 lun=0] - +~~~~ Use the sginfo -s &lt;device_name&gt; to identify disks with the same serial number on both management nodes, for example: On the primary management node: - + +~~~~ [root@primary ~]# sginfo -s /dev/sdb Serial Number '1T23043224 ' [root@primary ~]# - +~~~~ On the standby management node: - + +~~~~ [root@standby~]# sginfo -s /dev/sdb Serial Number '1T23043224 ' [root@standby ~]# - +~~~~ We can see that the /dev/sdb is a shared disk on both management nodes. In some cases, as with mirrored disks and when there is no matching of serial numbers between the two management nodes, multiple disks on a single server can have the same serial number, In these cases, format the disks, mount them on both management nodes, and then touch files on the disks to determine if they are shared between the management nodes. **2\. Create partitions on shared disks** After the shared disks are identified, create the partitions on the shared disks using fdisk command on the primary management node. Here is an example: - + +~~~~ fdisk /dev/sdc - +~~~~ Verify the partitions are created by running fdisk -l. **3\. Create file systems on shared disks** Run the mkfs.ext3 command on the primary management node to create file systems on the shared disk that will contain the xCAT data. For example: - + +~~~~ mkfs.ext3 -v /dev/sdc1 mkfs.ext3 -v /dev/sdc2 mkfs.ext3 -v /dev/sdc3 mkfs.ext3 -v /dev/sdc4 - +~~~~ If you place entries for the disk in /etc/fstab, which is not required, ensure that the entries do not have the system automatically mount the disk. @@ -1014,35 +1034,39 @@ **4\. Verify the file systems on the primary management node.** Verify the file systems could be mounted and written on the primary management node, here is an example: - + +~~~~ mount /dev/sdc1 /etc/xcat mount /dev/sdc2 /install mount /dev/sdc3 ~/.xcat mount /dev/sdc4 /db2database - +~~~~ After that, umount the file system on the primary management node: - + +~~~~ umount /etc/xcat umount /install umount ~/.xcat umount /db2database - +~~~~ **5\. Verify the file systems on the standby management node** On the standby management node, verify the file systems could be mounted and written. - + +~~~~ mount /dev/sdc1 /etc/xcat mount /dev/sdc2 /install mount /dev/sdc3 ~/.xcat mount /dev/sdc4 /db2database - +~~~~ You may get errors "mount: you must specify the filesystem type" or "mount: special device /dev/sdb1 does not exist" when trying to mount the file systems on the standby management node, this is caused by the missing devices files on the standby management node, run fidsk /dev/sdx and simply select "w write table to disk and exit" in the fdisk menu, then retry the mount. After that, umount the file system on the standby management node: - + +~~~~ umount /etc/xcat umount /install umount ~/.xcat umount /db2database - +~~~~ </pre> </div>Lissa VallettaThu, 24 Jul 2014 12:03:16 -0000https://sourceforge.net201a37d645189438c41528bc46d2a19887a23975https://sourceforge.net/p/xcat/wiki/Setup_HA_Mgmt_Node_With_Shared_Data/<div class="markdown_content"><pre>--- v24 +++ v25 @@ -46,23 +46,25 @@ **Note: Shared data itself needs high availability also, the shared data should not become a single point of failure.** -The configuration procedure will be quite different based on the shared data mechanism that will be used. Configuring these shared data mechanisms is beyond the scope of this documentation. The [Appendix A] gives an example of how to configure shared storage. After the shared data mechanism is configured, the following xCAT directory structure should be on the shared data, if this is done before xCAT is installed, you need to create the directories manually; if this is done after xCAT is installed, the directories need to be copied to the shared data. - +The configuration procedure will be quite different based on the shared data mechanism that will be used. Configuring these shared data mechanisms is beyond the scope of this documentation. The [Setup_HA_Mgmt_Node_With_Shared_Data/#appendix-a-configure-shared-disks](Setup_HA_Mgmt_Node_With_Shared_Data/#appendix-a-configure-shared-disks) gives an example of how to configure shared storage. After the shared data mechanism is configured, the following xCAT directory structure should be on the shared data, if this is done before xCAT is installed, you need to create the directories manually; if this is done after xCAT is installed, the directories need to be copied to the shared data. + +~~~~ /etc/xcat /install ~/.xcat /&lt;dbdirectory&gt; (For mysql, the database directory is /var/lib/mysql; for postgresql, the database directory is /var/lib/pgsql; for DB2, the database directory is specified with the site attribute databaseloc; for sqlite, the database directory is /etc/xcat, already listed above. ) - +~~~~ Here is an example of how to make directories be shared data through NFS: - + +~~~~ mount -o rw &lt;nfssvr&gt;:/dir1 /etc/xcat mount -o rw &lt;nfssvr&gt;:/dir2 /install mount -o rw &lt;nfssvr&gt;:/dir3 ~/.xcat mount -o rw &lt;nfssvr&gt;:/dir4 /&lt;dbdirectory&gt; - +~~~~ **Note: if you need to setup high availability for some other applications, like the HPC software stack, between the two xCAT management nodes, the applications data should be on the shared data.** @@ -72,15 +74,18 @@ 2\. Set up a "Virtual IP address". The xcatd daemon should be addressable with the same Virtual IP address, regardless of which management node it runs on. The same Virtual IP address will be configured as an alias IP address on the management node (primary and standby) that the xcatd runs on. The Virtual IP address can be any unused ip address that all the compute nodes and service nodes could reach. Here is an example on how to configure Virtual IP on AIX and Linux: -On AIX: - + +On AIX: + +~~~~ ifconfig en0 9.114.47.97 netmask 255.255.255.192 firstalias - - -On Linux: - +~~~~ + +On Linux: + +~~~~ ifconfig eth0:0 9.114.47.97 netmask 255.255.255.192 - +~~~~ The option "firstalias" will configure the Virtual IP ahead of the interface ip address, since ifconfig will not make the ip address configuration be persistent through reboots, so the Virtual IP address needs to be re-configured right after the management node is rebooted. This non-persistent Virtual IP address is designed to avoid ip address conflict when the crashed previous primary management is recovered with the Virtual IP address configured. @@ -91,35 +96,40 @@ 4\. Install xCAT. The procedure described in [XCAT_iDataPlex_Cluster_Quick_Start] could be used for the xCAT setup on the primary management node. 5\. Check the site table master and nameservers and network tftpserver attribute is the Virtual ip - + +~~~~ lsdef -t site - +~~~~ If not correct: - + +~~~~ chdef -t site master=9.114.47.97 chdef -t site nameservers=9.114.47.97 chdef -t network tftpserver=9.114.47.97 - +~~~~ Add the two management nodes into policy table - + +~~~~ tabdump policy "1.2","aixmn1",,,,,,"trusted",, "1.3","aixmn2",,,,,,"trusted",, - +~~~~ 6\. (Optional) DB2 only, change the databaseloc in site table - + +~~~~ chdef -t site databaseloc=/dbdirectory - +~~~~ 7\. Install and configure database. Refer to the doc [Choosing_the_Database] to configure the database on the xCAT management node. Verify xcat is running on correct database by running: - + +~~~~ lsxcatd -a - +~~~~ 8\. Backup the xCAT database tables for the current configuration on standby management node, using command dumpxCATdb -p &lt;yourbackupdir&gt;. @@ -128,59 +138,66 @@ 10\. Stop the xcatd daemon and some related network services from starting on reboot: On AIX: - + +~~~~ stopsrc -s xcatd chitab "xcatd:2:off:/opt/xcat/sbin/restartxcatd &gt; /dev/console 2&gt;&amp;1" stopsrc -s conserver chitab "conserver:2:off:/usr/bin/startsrc -s conserver &gt; /dev/console 2&gt;&amp;1" - +~~~~ Make sure that in /etc/rc.tcpip,the following lines are commented out. - + +~~~~ grep named /etc/rc.tcpip #start /usr/sbin/named "$src_running" grep dhcpsd /etc/rc.tcpip #start /usr/sbin/dhcpsd "$src_running" - - -On Linux: - +~~~~ + +On Linux: + +~~~~ service xcatd stop chkconfig --level 345 xcatd off service conserver off chkconfig --level 2345 conserver off service dhcpd stop chkconfig --level 2345 dhcpd off - +~~~~ 11\. Stop Database and prevent the database from auto starting at boot time Use mysql as an example: On AIX: - + +~~~~ /usr/local/mysql/bin/mysqladmin -u root -p shutdown In /etc/inittab mysql:2:off:/usr/local/mysql/bin/mysqld_safe --user=mysql &amp; - - -On Linux: - +~~~~ + +On Linux: + +~~~~ service mysqld stop chkconfig mysqld off - +~~~~ 12\. (Optional) If DFM is being used for hardware control capabilities, install DFM package,Setup xCAT to communicate directly to the System P server's service processor. - + +~~~~ xCAT-dfm RPM ISNM-hdwr_svr RPM (linux) isnm.hdwr_svr installp package (AIX) - + +~~~~ 13\. If there is any node that is already managed by the Management Node,change the noderes table tftpserver &amp; xcatmaster &amp; nfsserver attributes to the Virtual ip @@ -199,21 +216,24 @@ Also configure the Virtual IP address during this setup. On AIX: - + +~~~~ ifconfig en0 9.114.47.97 netmask 255.255.255.192 firstalias - - -On Linux: - +~~~~ + +On Linux: + +~~~~ ifconfig eth0:0 9.114.47.97 netmask 255.255.255.192 - +~~~~ 4\. Install xCAT. The procedure described in [XCAT_iDataPlex_Cluster_Quick_Start] can be used for the xCAT setup on the standby management node. The database system on the standby management node must be the same as the one running on the primary management node. 5\. (Optional) DB2 only, change the databaseloc in site table to be the same as the the primary management node. - + +~~~~ chdef -t site databaseloc=/dbdirectory - +~~~~ 6\. (Optional) DB2 only, check databaseloc directory @@ -225,50 +245,56 @@ 7\. (Optional) DB2 only, setup the xcatdb id and group and install database. When setting up database on the Standby, you must make sure that the xcatdb userid groupid and password match what is on the Primary management node. This id is your DB2 database instance and will be mounting the database from the shared data when the Standby takes over. Look up the xcatdb user id number and the xcatdb group id number on the Primary management node. You can use lsuser, lsgroup for AIX , or just look in /etc/passwd and /etc/group. On AIX: - + +~~~~ mkgroup -a id=&lt;xcatdb group="" id="" number="" from="" the="" Primary="" management="" node=""&gt; xcatdb mkuser pgrp='xcatdb' home='&lt;xcatdb home="" directory="" from="" Primary="" management="" node=""&gt;' shell='/bin/ksh' id='&lt;xcatdb user="" id="" number="" from="" Primary="" management="" node=""&gt;' xcatdb - - -On Linux: - +~~~~ + +On Linux: + +~~~~ groupadd -g &lt;group id="" number="" from="" the="" Primary="" management="" node=""&gt; xcatdb useradd -d &lt;xcatdb home="" directory="" from="" the="" Primary="" management="" node=""&gt; -g xcatdb -u &lt;uid number="" from="" the="" Primary="" management="" node=""&gt; -m -s /bin/bash xcatdb - +~~~~ Set the xcatdb password On AIX (change passwd to match the password on the Primary management node) - + +~~~~ chpasswd -c xcatdb:&lt;passwd&gt; ctl-D - +~~~~ On Linux ( change passwd to match the password on the Primary management node) - chpasswd +~~~~ + chpasswd xcatdb:&lt;passwd&gt; ctl-D - +~~~~ Note: we use the chpasswd command so the password will not have to be changed the first time we use the xcatdb userid. Install and configure DB2 software on the standby management node using the instructions in [Setting_Up_DB2_as_the_xCAT_DB]. Install DB2 and run db2sqlsetup to setup the xCAT database. Verify if xCAT is running correctly with DB2 on the standby management node by running: - + +~~~~ lsxcatd -a - +~~~~ 8\. (Optional) DFM only, Install DFM package When installing and configuring DFM on the standby management node,you should follow the document in [XCAT_Power_775_Hardware_Management]. - + +~~~~ xCAT-dfm RPM ISNM-hdwr_svr RPM (linux) isnm.hdwr_svr installp package (AIX) - +~~~~ 9\. Setup hostname resolution between the primary management node and standby management node. Make sure the primary management node can resolve the hostname of the standby management node, and vice versa. @@ -287,64 +313,70 @@ To setup ntp on the management nodes on AIX: Update the /etc/ntp.conf file with a valid ntp server. - + +~~~~ stopsrc -s xntpd startsrc -s xntpd - +~~~~ Use ntpq -p to show the peer status of the ntp server, should see * to left of server after successful association with server is established. 12\. Stop the xcatd daemon and related network services from starting on reboot: On AIX: - + +~~~~ stopsrc -s xcatd chitab "xcatd:2:off:/opt/xcat/sbin/restartxcatd &gt; /dev/console 2&gt;&amp;1" stopsrc -s conserver chitab "conserver:2:off:/usr/bin/startsrc -s conserver &gt; /dev/console 2&gt;&amp;1" - +~~~~ Make sure that in /etc/rc.tcpip,the following lines are commented out. - + +~~~~ grep named /etc/rc.tcpip #start /usr/sbin/named "$src_running" grep dhcpsd /etc/rc.tcpip #start /usr/sbin/dhcpsd "$src_running" - - -On Linux: - +~~~~ + +On Linux: + +~~~~ service xcatd stop chkconfig --level 345 xcatd off service conserver off chkconfig --level 2345 conserver off service dhcpd stop chkconfig --level 2345 dhcpd off - +~~~~ 13\. Stop Database and prevent the database from auto starting at boot time Use mysql as an example: On AIX: - + +~~~~ /usr/local/mysql/bin/mysqladmin -u root -p shutdown In /etc/inittab mysql:2:off:/usr/local/mysql/bin/mysqld_safe --user=mysql &amp; - - -On Linux: - +~~~~ + +On Linux: + +~~~~ service mysqld stop chkconfig mysqld off - +~~~~ 14\. Backup the xCAT database tables for the current configuration on standby management node, using command dumpxCATdb -p &lt;yourbackupdir&gt;. @@ -353,14 +385,16 @@ 16\. Remove the Virtual Alias IP. On AIX: - + +~~~~ ifconfig en0 delete 9.114.47.97 - - -On Linux: - +~~~~ + +On Linux: + +~~~~ ifconfig eth0:0 0.0.0.0 0.0.0.0 - +~~~~ @@ -376,9 +410,10 @@ The xCAT SSL credentials reside in the directories /etc/xcat/ca, /etc/xcat/cert and $HOME/.xcat/. The ssh host keys that xCAT generates to be placed on the compute nodes are in the directory /etc/xcat/hostkeys. These directories are on the shared data. In addition the ssh root keys in the management node's root home directory (in ~/.ssh) must be kept in sync between the primary management node and standby management node. Only sync the key files and not the authorized_key file. These keys will seldom change, so you can just do it manually when they do, or setup a cron entry like this sample: - + +~~~~ 0 1 * * * /usr/bin/rsync -Lprgotz $HOME/.ssh/id* aixmn2:$HOME/.ssh/ - +~~~~ Now go to the Standby node and add the Primary's id_rsa.pub to the Standby's authorized_keys file. @@ -390,14 +425,15 @@ - +~~~~ 0 2 * * * /usr/bin/rsync -Lprogtz /etc/hosts aixmn2:/etc/ - +~~~~ ### **Additional Customization Files and Production files** Besides the files mentioned above, there may be some additional customization files and production files that need to be copied over to the standby management node, depending on your local unique requirements. You should always try to keep the standby management node as an identical clone of the primary management node. Here are some example files that can be considered: - + +~~~~ /.profile /.rhosts /etc/auto_master @@ -416,7 +452,7 @@ /etc/services /etc/inittab (and more) - +~~~~ Note: @@ -445,96 +481,108 @@ ### Take down the Current Primary Management Node Starting with xCAT 2.8.2, xCAT ships a sample script /opt/xcat/share/xcat/hamn/deactivate-mn to make the machine be a standby management node. Before using this script, you need to review the script carefully and make updates accordingly, here is an example of how to use this script: - + +~~~~ /opt/xcat/share/xcat/hamn/deactivate-mn -i eth1:2 -v 9.114.47.97 - +~~~~ **On the current primary management node:** If the management node is still available and running the cluster, perform the following steps to shutdown. **1\. (DFM only) Remove connections from CEC and Frame. ** - + +~~~~ rmhwconn cec,frame rmhwconn cec,frame -T fnm - +~~~~ **2\. Stop the xCAT daemon.** Note xCAT must be stopped on all Service Nodes also, and LL if using the database. On AIX: - + +~~~~ stopsrc -s xcatd stopsrc -s dhcpsd stopsrc -s conserver stopsrc -s hdwr_svr stopsrc -s named - - -On Linux: - +~~~~ + +On Linux: + +~~~~ service xcatd stop service dhcpd stop - +~~~~ **3\. unexport the xCAT NFS directories** The exported xCAT NFS directories will prevent the shared data partitions from being unmounted, so the exported xCAT NFS directories should be unmounted before failover. - + +~~~~ exportfs -ua - +~~~~ **4\. Stop database** Use mysql as an example: - + +~~~~ service mysqld stop - +~~~~ **5\. unmount shared data** All the file systems on the shared data need to be unmounted to make the previous standby management be able to mount the file systems on the shared data. Here is an example: - + +~~~~ umount /etc/xcat umount /install umount ~/.xcat umount /db2database - +~~~~ When trying to umount the file systems, if there are some processes that are accessing the files and directories on the file systems, you will get "Device busy" error. The following commands can be used to check which progresses are accessing the file systems on AIX: - + +~~~~ fuser -uxc &lt;directory_name&gt; - +~~~~ Then stop or kill all the processes that are accessing the shared data file systems and retry the unmount. **6\. (Optional, AIX and shared disk only)varyoff volume group** - + +~~~~ varyoffvg xcatvg - +~~~~ **7\. Unconfigure Virtual IP** On AIX: - + +~~~~ ifconfig en0 delete 9.114.47.97 - - -On Linux: - +~~~~ + +On Linux: + +~~~~ ifconfig eth0:0 0.0.0.0 0.0.0.0 - +~~~~ If the ifconfig command has been added to rc.local, remove it from rc.local. ### Bring up the New Primary Management Node Starting with xCAT 2.8.2, xCAT ships a sample script /opt/xcat/share/xcat/hamn/activate-mn to make the machine be a primary management node. Before using this script, you need to review the script carefully and make updates accordingly, here is an example of how to use this script: - + +~~~~ /opt/xcat/share/xcat/hamn/activate-mn -i eth1:2 -v 9.114.47.97 -m 255.255.255.0 - +~~~~ **On the new primary management node:** @@ -542,29 +590,33 @@ **1\. Configure Virtual IP** On AIX: - + +~~~~ ifconfig en0 9.114.47.97 netmask 255.255.255.192 firstalias - - -On Linux: - +~~~~ + +On Linux: + +~~~~ ifconfig eth0:0 9.114.47.97 netmask 255.255.255.192 - +~~~~ You can put the ifconfig command into rc.local to make the Virtual IP be persistent after reboot. **2\. (Optional,AIX and shared disk only) varyon volume group** - + +~~~~ varyonvg xcatvg - +~~~~ **3\. Mount shared data** - + +~~~~ mount /etc/xcat mount /install mount /.xcat mount /db2database - +~~~~ @@ -572,36 +624,40 @@ **4\. Start database** Use mysql as an example: - + +~~~~ service mysql start - +~~~~ **5\. Start the daemons.** On AIX: - + +~~~~ startsrc -s dhcpsd restartxcatd startsrc -s hdwr_svr startsrc -s conserver startsrc -s named - +~~~~ On Linux: - + +~~~~ service dhcpd start service xcatd start service hdwr_svr start service conserver start - +~~~~ **6\. (DFM only) Setup connection for CEC and Frame** + mkhwconn cec,frame -t mkhwconn cec,frame -t -T fnm chnwm -a - +~~~~ **7\. Setup network services and conserver** </pre> </div>Lissa VallettaThu, 24 Jul 2014 11:50:37 -0000https://sourceforge.net252568783a39c5355b1fa0c09bf2d574e3dcec73https://sourceforge.net/p/xcat/wiki/Setup_HA_Mgmt_Node_With_Shared_Data/<div class="markdown_content"><pre>--- v23 +++ v24 @@ -51,17 +51,17 @@ /etc/xcat /install ~/.xcat - /&lt;dbdirectory&gt; (For mysql, the database directory is /var/lib/mysql; for postgresql, the database directory is /var/lib/pgsql; + /&lt;dbdirectory&gt; (For mysql, the database directory is /var/lib/mysql; for postgresql, the database directory is /var/lib/pgsql; for DB2, the database directory is specified with the site attribute databaseloc; for sqlite, the database directory is /etc/xcat, already listed above. ) Here is an example of how to make directories be shared data through NFS: - mount -o rw &lt;nfssvr&gt;:/dir1 /etc/xcat - mount -o rw &lt;nfssvr&gt;:/dir2 /install - mount -o rw &lt;nfssvr&gt;:/dir3 ~/.xcat - mount -o rw &lt;nfssvr&gt;:/dir4 /&lt;dbdirectory&gt; + mount -o rw &lt;nfssvr&gt;:/dir1 /etc/xcat + mount -o rw &lt;nfssvr&gt;:/dir2 /install + mount -o rw &lt;nfssvr&gt;:/dir3 ~/.xcat + mount -o rw &lt;nfssvr&gt;:/dir4 /&lt;dbdirectory&gt; **Note: if you need to setup high availability for some other applications, like the HPC software stack, between the two xCAT management nodes, the applications data should be on the shared data.** @@ -121,7 +121,7 @@ lsxcatd -a -8\. Backup the xCAT database tables for the current configuration on standby management node, using command dumpxCATdb -p &lt;yourbackupdir&gt;. +8\. Backup the xCAT database tables for the current configuration on standby management node, using command dumpxCATdb -p &lt;yourbackupdir&gt;. 9\. Setup a crontab to backup the database each night by running dumpxCATdb and storing the backup to some filesystem not on the shared data. @@ -226,14 +226,14 @@ On AIX: - mkgroup -a id=&lt;xcatdb group id number from the Primary management node&gt; xcatdb - mkuser pgrp='xcatdb' home='&lt;xcatdb home directory from Primary management node&gt;' shell='/bin/ksh' id='&lt;xcatdb user id number from Primary management node&gt;' xcatdb - - -On Linux: - - groupadd -g &lt;group id number from the Primary management node&gt; xcatdb - useradd -d &lt;xcatdb home directory from the Primary management node&gt; -g xcatdb -u &lt;uid number from the Primary management node&gt; -m -s /bin/bash xcatdb + mkgroup -a id=&lt;xcatdb group="" id="" number="" from="" the="" Primary="" management="" node=""&gt; xcatdb + mkuser pgrp='xcatdb' home='&lt;xcatdb home="" directory="" from="" Primary="" management="" node=""&gt;' shell='/bin/ksh' id='&lt;xcatdb user="" id="" number="" from="" Primary="" management="" node=""&gt;' xcatdb + + +On Linux: + + groupadd -g &lt;group id="" number="" from="" the="" Primary="" management="" node=""&gt; xcatdb + useradd -d &lt;xcatdb home="" directory="" from="" the="" Primary="" management="" node=""&gt; -g xcatdb -u &lt;uid number="" from="" the="" Primary="" management="" node=""&gt; -m -s /bin/bash xcatdb Set the xcatdb password @@ -241,14 +241,14 @@ On AIX (change passwd to match the password on the Primary management node) chpasswd -c - xcatdb:&lt;passwd&gt; + xcatdb:&lt;passwd&gt; ctl-D On Linux ( change passwd to match the password on the Primary management node) chpasswd - xcatdb:&lt;passwd&gt; + xcatdb:&lt;passwd&gt; ctl-D @@ -346,7 +346,7 @@ chkconfig mysqld off -14\. Backup the xCAT database tables for the current configuration on standby management node, using command dumpxCATdb -p &lt;yourbackupdir&gt;. +14\. Backup the xCAT database tables for the current configuration on standby management node, using command dumpxCATdb -p &lt;yourbackupdir&gt;. 15\. Change the hostname back to the original hostname. @@ -505,7 +505,7 @@ When trying to umount the file systems, if there are some processes that are accessing the files and directories on the file systems, you will get "Device busy" error. The following commands can be used to check which progresses are accessing the file systems on AIX: - fuser -uxc &lt;directory_name&gt; + fuser -uxc &lt;directory_name&gt; Then stop or kill all the processes that are accessing the shared data file systems and retry the unmount. </pre> </div>BruceTue, 08 Jul 2014 12:12:45 -0000https://sourceforge.net655d68211d514b2b0b43ca9029712bec52f9d6b4