Recent changes to Rolling_Update_Support

Rolling_Update_Support modified by

Tue, 21 Apr 2015 16:03:04 -0000

--- v80
+++ v81
@@ -1,3 +1,7 @@
+## THE xCAT ROLLING UPDATE SUPPORT IS OBSOLETE ##
+The xCAT Rolling Update support requires the Tivoli Workload Scheduler LoadLeveler to schedule the nodes for update.  LoadLeveler has been superseded by LSF in the IBM HPC Software stack, so this xCAT function is no longer supported.  
+
+
 ![](http://sourceforge.net/p/xcat/wiki/XCAT_Documentation/attachment/Official-xcat-doc.png)

 [TOC]

Rolling_Update_Support modified by Lissa Valletta

Lissa Valletta — Tue, 09 Sep 2014 18:30:11 -0000

--- v79
+++ v80
@@ -97,7 +97,7 @@

 Stanza input is specified as: 

-_**keyword**_ **=** _**value**_
+**keyword** **=** **value**

 with one keyword per line. Unless otherwise noted in the descriptions below, if multiple stanza lines are specified for the same keyword, only the FIRST entry will be used and all others will be ignored. 

@@ -141,7 +141,7 @@
 **job_dir**

-**updateall_nodes = **xCAT nodereange
+**updateall_nodes = **xCAT noderange

 Used with **updateall=yes**. The xCAT nodereange specifies the list of nodes that are to be included in this rolling update request (see the xCAT **noderange** man page).. All nodes must be active in the job scheduler in order to be updated. 

@@ -262,13 +262,13 @@
 The default is 1. 

-**translatenames = **_noderange_:|_pattern_|_replacement_| 
-
-**translatenames = **_noderange_:/_pattern_/_replacement_/ 
+**translatenames = **noderange:|pattern|replacement| 
+
+**translatenames = **noderange:/pattern/replacement/ 

 (optional) If your scheduler will be using names for nodes that are different from xCAT node names (e.g. the scheduler is using a different administrative network), you will need to tell xCAT how to translate from xCAT node names to the node names registered with your scheduler. 

-_pattern_ and _replacement_ are perl regular expressions to be performed on the node names in _noderange_. See the xcatdb man page for more details on using regular expressions. Multiple translatenames stanzas are allowed. If an xCAT nodename exists in more than one _noderange_, the last translated value will be used. 
+pattern and replacement are perl regular expressions to be performed on the node names in noderange. See the xcatdb man page for more details on using regular expressions. Multiple translatenames stanzas are allowed. If an xCAT nodename exists in more than one noderange, the last translated value will be used. 

 For example, to translate names of the form "bb1s1" to "bb1sn1": translatenames=service:|bb(\d+)s(\d+)|bb($1)sn($2)|

Rolling_Update_Support modified by Lissa Valletta

Lissa Valletta — Tue, 09 Sep 2014 18:27:57 -0000

--- v78
+++ v79
@@ -26,7 +26,7 @@
 Out-of-band commands can be used to perform operations that require the nodes to be powered down, such as firmware updates.

-_Restrictions:_
+Restrictions:

   1. Support for including infrastructure nodes such as Service Nodes and GPFS I/O nodes in update groups is released as "experimental only". All required function has been coded and unit tested by xCAT development, but this support has not been tested in a formal Functional Verification Test or System Test environment. 
   2. For Power 775 clusters, the GPFS I/O nodes and the compute nodes are typically installed as statelite nodes, mounting persistent directories through soft NFS mounts served by their service nodes. While it may be possible that a compute node can continue to operate when its service node goes down, the continued operation of the GPFS I/O nodes and the availability of your GPFS filesystems may be compromised. The xCAT rolling update solution will not be supported for service nodes (which includes firmware updates of CECs in which a service node resides) until a high-availability solution is in place for these clusters, or if you have implemented a cluster solution that allows GPFS filesystems to remain operational when any service node goes down. 
@@ -104,24 +104,24 @@

 Valid keywords are: 

-**scheduler = **_scheduler_
-
-where _scheduler_ is the job scheduler used to submit the rolling update jobs. Currently only "loadleveler" is supported. 
-
-  
-**scheduser = **_scheduser_
-
-where _scheduser_ is the userid with authority to submit scheduler jobs. Note that LoadLeveler does not allow reservation jobs to be submitted by the root userid. 
-
-  
-**oldfeature = **_feature value_
-
-(optional) where _feature value _is an existing LoadLeveler feature value that is set for the nodes being updated. xCAT will remove this value from the LoadLeveler machine definition after the update has completed. 
-
-  
-**newfeature = **_feature value_
-
-(optional) where _feature value _is a new LoadLeveler feature value to be set for the nodes being updated. xCAT will add this value to the LoadLeveler machine definition after the update has completed. This can be useful for users that wish to schedule jobs that can only be run on nodes that have been updated. 
+**scheduler = **scheduler
+
+where scheduler is the job scheduler used to submit the rolling update jobs. Currently only "loadleveler" is supported. 
+
+  
+**scheduser = **scheduser
+
+where scheduser is the userid with authority to submit scheduler jobs. Note that LoadLeveler does not allow reservation jobs to be submitted by the root userid. 
+
+  
+**oldfeature = **feature value
+
+(optional) where feature value is an existing LoadLeveler feature value that is set for the nodes being updated. xCAT will remove this value from the LoadLeveler machine definition after the update has completed. 
+
+  
+**newfeature = **feature value
+
+(optional) where feature value is a new LoadLeveler feature value to be set for the nodes being updated. xCAT will add this value to the LoadLeveler machine definition after the update has completed. This can be useful for users that wish to schedule jobs that can only be run on nodes that have been updated. 

 **updateall = yes | no**
@@ -141,28 +141,28 @@
 **job_dir**

-**updateall_nodes = **_xCAT nodereange_
-
-Used with **updateall=yes**. The _xCAT nodereange _specifies the list of nodes that are to be included in this rolling update request (see the xCAT **noderange** man page).. All nodes must be active in the job scheduler in order to be updated. 
-
-  
-**updateall_nodecount = **_numeric value_
-
-Used with updateall=yes. The _numeric value _specifies the number of nodes that will be reserved at one time in the scheduler and updated together. The smaller the number, the more scheduler reservation jobs that will be submitted. 
+**updateall_nodes = **xCAT nodereange
+
+Used with **updateall=yes**. The xCAT nodereange specifies the list of nodes that are to be included in this rolling update request (see the xCAT **noderange** man page).. All nodes must be active in the job scheduler in order to be updated. 
+
+  
+**updateall_nodecount = **numeric value
+
+Used with updateall=yes. The numeric value specifies the number of nodes that will be reserved at one time in the scheduler and updated together. The smaller the number, the more scheduler reservation jobs that will be submitted. 

 NOTE: LoadLeveler performance decreases with large numbers of reservations. Do not set this value so low that you will exceed the maximum number of reservations allowed for your cluster or that you will degrade LL performance for your production jobs. You must also ensure that the LL MAX_RESERVATIONS setting is large enough to handle all the reservations that will be created. 

-**updategroup = **_name(noderange)_
-
-For standard updates, at least one **updategroup** or **mapgroups** stanza must be specified. The _name _specifies the name to be assigned to this update group. The _noderange _is an xCAT noderange that specifies the list of nodes that will be included in this update group (see the xCAT **noderange** man page). 
+**updategroup = **name(noderange)
+
+For standard updates, at least one **updategroup** or **mapgroups** stanza must be specified. The name specifies the name to be assigned to this update group. The noderange is an xCAT noderange that specifies the list of nodes that will be included in this update group (see the xCAT **noderange** man page). 

 Multiple **updategroup** stanzas may be supplied, one for each group of nodes to be updated. 

-**mapgroups = **_nodegroup range_
-
-For standard updates, at least one **updategroup** or **mapgroups** stanza must be specified. . The _nodegroup range _specifies a list or range of xCAT nodegroup names. This field is processed in the same way the xCAT noderange processing works for node names, except that it will generate a list of xCAT nodegroup names. Each nodegroup in the list will become its own update group for this rolling update request, with the update group name set to the nodegroup name. 
+**mapgroups = **nodegroup range
+
+For standard updates, at least one **updategroup** or **mapgroups** stanza must be specified. . The nodegroup range specifies a list or range of xCAT nodegroup names. This field is processed in the same way the xCAT noderange processing works for node names, except that it will generate a list of xCAT nodegroup names. Each nodegroup in the list will become its own update group for this rolling update request, with the update group name set to the nodegroup name. 

 Multiple **mapgroups** stanzas may be supplied. 

@@ -171,9 +171,9 @@
 mapgroups=block[01-10] 

-**mutex = **_updategroup,updategroup,..._
-
-(optional) Mutual exclusion for standard updates. The comma-delimited list of _updategroup_ names specify which update groups are mutually exclusive and must not be updated at the same time in order to maintain active resources within the cluster. By default, only one updategroup listed in the entry will be updated at a time (see **mutex_count** below to change this default). 
+**mutex = **updategroup,updategroup,...
+
+(optional) Mutual exclusion for standard updates. The comma-delimited list of updategroup names specify which update groups are mutually exclusive and must not be updated at the same time in order to maintain active resources within the cluster. By default, only one updategroup listed in the entry will be updated at a time (see **mutex_count** below to change this default). 

 You may list multiple **mutex** stanzas to identify different sets of mutual exclusion. 

@@ -182,9 +182,9 @@
 mutex=ns1,ns2 

-**mutex = **_updategroup range,updategroup range,..._
-
-(optional) For standard updates. The comma-delimited list of _updategroup ranges_ will be processed in the same way the xCAT noderange processing works for node names, except that it will generate a list of rolling update update group names. The first name in each range is paired together to make a mutual exclusion list, the second name in each range is paired together, etc. 
+**mutex = **updategroup range,updategroup range,...
+
+(optional) For standard updates. The comma-delimited list of updategroup ranges will be processed in the same way the xCAT noderange processing works for node names, except that it will generate a list of rolling update update group names. The first name in each range is paired together to make a mutual exclusion list, the second name in each range is paired together, etc. 

 For example, the following single entry: 

@@ -199,7 +199,7 @@
 mutex=block3a,block3b,block3c 

-**nodegroup_mutex = **_nodegroup name_
+**nodegroup_mutex = **nodegroup name

 (optional) For standard updates. Mutual exclusion for any nodes in this xCAT nodegroup. For each updategroup listed above, if any nodes in that group are a member of this xCAT nodegroup, add it to the mutex entry. 

@@ -228,7 +228,7 @@
 By default, only one of these update groups will be updated at a time unless a different mutex_count is specified for this stanza (see mutex_count below). 

-**nodegroup_mutex = **_nodegroup name range_
+**nodegroup_mutex = **nodegroup name range

 (optional) For standard updates. Specify multiple nodegroup_mutex statements with a single stanza. The nodegroup name range is expanded to list of xCAT nodegroup names. This then becomes equivalent to multiple nodegroup_mutex stanzas. 

@@ -247,9 +247,9 @@
 Which, in turn, would be evaluated to create the correct mutex statements following the nodegroup_mutex processing described above. 

-**mutex_count = **_numeric value _
-
-(optional) where _numeric value _is the number of update groups in a mutex statement that can be run at the same time. For example, if you have: 
+**mutex_count = **numeric value 
+
+(optional) where numeric value is the number of update groups in a mutex statement that can be run at the same time. For example, if you have: 

 mutex=c1,c2,c3,c4

Rolling_Update_Support modified by

Tue, 19 Aug 2014 18:13:33 -0000

Rolling_Update_Support modified by

Tue, 19 Aug 2014 18:13:01 -0000

--- v76
+++ v77
@@ -80,7 +80,7 @@

   * The LoadLeveler userid specified in the xCAT rollupdate input **scheduser** stanza must be authorized to run the xCAT "runrollupdate" command. See the xCAT wiki how-to [Granting_Users_xCAT_privileges](Granting_Users_xCAT_privileges) to set this up. This includes creating SSL certificates for the user and changing the xCAT policy table. 
-  * Loadleveler settings need to be able to handle flexible reservations. See chapter on [LoadLeveler Settings](#LoadLeveler_Settings) for a listing of additional requirements. 
+  * Loadleveler settings need to be able to handle flexible reservations. See chapter on [LoadLeveler Settings](#loadLeveler-settings) for a listing of additional requirements. 
   * The root userid on the xCAT management node must have LOADL_ADMIN privileges. 
   * You have created the new OS image that will be loaded onto the nodes and run all of the necessary xCAT commands such that when the node is rebooted, it will load your updated image. 

@@ -495,7 +495,7 @@
   2. If you are using Linux service nodes that do not have the /install directory mounted, be sure to copy the new image down to your service nodes. 
   3. Run all necessary xCAT commands to prepare to boot the nodes with the new OS image (nodeset, mkdsklsnode). 
   4. If you wish to have your LoadLeveler users be able to run jobs on either just old nodes or just updated nodes during the rolling update process, you should set a LL FEATURE value for all of the machine definitions in your cluster. You can then specify this as an **oldfeature** stanza value in your rollupdate command input. Also, be sure to specify a **newfeature** stanza value to identify updated nodes. Your users can use these feature values in their job "requirements field. 
-  5. Create your rollupdate command input stanza file. See the chapter on [rollupdate Input Stanza File](#rollupdate_Input_Stanza_File)for details. 
+  5. Create your rollupdate command input stanza file. See the chapter on [rollupdate Input Stanza File](#rollupdate-input-stanza-file)for details. 
   6. (Optional) If you wish to view the progress of the rollupdate process, in a separate window, open and monitor the rollupdate log: `tail -f /var/log/xcat/rollupdate.log`xCAT appends to the log file with each rollupdate run, so you may wish to move or remove the log file before a new rollupdate process. 
   7. Initiate the rollupdate process: `cat <your stanza file> | rollupdate --verbose`If you want to test your input first to make sure it is what you want, and to view the LL reservation job command files and other data files generated by xCAT, run the command with the test option: `cat <your stanza file> | rollupdate -test --verbose`The output files will be placed in the directory you specified in the **jobdir** stanza. The verbose keyword is optional, but allows you to view detailed progress.

Rolling_Update_Support modified by

Tue, 19 Aug 2014 18:11:07 -0000

--- v75
+++ v76
@@ -36,7 +36,7 @@

 The following prerequisites must be set up before running the xCAT rolling update support: 

-  * LoadLeveler must be installed and running on all of the cluster nodes that will be updated. See [Setting_up_the_IBM_HPC_Stack_in_an_xCAT_Cluster] for information on installing and configuring LoadLeveler in an xCAT cluster. It is possible to run updates for some nodes that are not active in LoadLeveler; however to use the full rolling update support, LL should be active. See the stanza input entries for the standard update method with **update_if_down=yes**. 
+  * LoadLeveler must be installed and running on all of the cluster nodes that will be updated. See [Setting_up_the_IBM_HPC_Stack_in_an_xCAT_Cluster](Setting_up_the_IBM_HPC_Stack_in_an_xCAT_Cluster) for information on installing and configuring LoadLeveler in an xCAT cluster. It is possible to run updates for some nodes that are not active in LoadLeveler; however to use the full rolling update support, LL should be active. See the stanza input entries for the standard update method with **update_if_down=yes**. 
   * LoadLeveler must be running with the database configuration option, using xCAT's database. 
   * The simplest configuration for using xCAT with LoadLeveler for rolling updates is to configure the LoadLeveler central manager on your xCAT management node. 
   * If the xCAT management node is not the LoadLeveler central manager, you must do the following: 
@@ -79,8 +79,8 @@
       SCHEDD_HOST=true

-  * The LoadLeveler userid specified in the xCAT rollupdate input **scheduser** stanza must be authorized to run the xCAT "runrollupdate" command. See the xCAT wiki how-to [Granting_Users_xCAT_privileges] to set this up. This includes creating SSL certificates for the user and changing the xCAT policy table. 
-  * Loadleveler settings need to be able to handle flexible reservations. See chapter on [#LoadLeveler_Settings] for a listing of additional requirements. 
+  * The LoadLeveler userid specified in the xCAT rollupdate input **scheduser** stanza must be authorized to run the xCAT "runrollupdate" command. See the xCAT wiki how-to [Granting_Users_xCAT_privileges](Granting_Users_xCAT_privileges) to set this up. This includes creating SSL certificates for the user and changing the xCAT policy table. 
+  * Loadleveler settings need to be able to handle flexible reservations. See chapter on [LoadLeveler Settings](#LoadLeveler_Settings) for a listing of additional requirements. 
   * The root userid on the xCAT management node must have LOADL_ADMIN privileges. 
   * You have created the new OS image that will be loaded onto the nodes and run all of the necessary xCAT commands such that when the node is rebooted, it will load your updated image. 

@@ -495,7 +495,7 @@
   2. If you are using Linux service nodes that do not have the /install directory mounted, be sure to copy the new image down to your service nodes. 
   3. Run all necessary xCAT commands to prepare to boot the nodes with the new OS image (nodeset, mkdsklsnode). 
   4. If you wish to have your LoadLeveler users be able to run jobs on either just old nodes or just updated nodes during the rolling update process, you should set a LL FEATURE value for all of the machine definitions in your cluster. You can then specify this as an **oldfeature** stanza value in your rollupdate command input. Also, be sure to specify a **newfeature** stanza value to identify updated nodes. Your users can use these feature values in their job "requirements field. 
-  5. Create your rollupdate command input stanza file. See the chapter on [3](#rollupdate_Input_Stanza_File)[ rollupdate Input Stanza File](#rollupdate_Input_Stanza_File)for details. 
+  5. Create your rollupdate command input stanza file. See the chapter on [rollupdate Input Stanza File](#rollupdate_Input_Stanza_File)for details. 
   6. (Optional) If you wish to view the progress of the rollupdate process, in a separate window, open and monitor the rollupdate log: `tail -f /var/log/xcat/rollupdate.log`xCAT appends to the log file with each rollupdate run, so you may wish to move or remove the log file before a new rollupdate process. 
   7. Initiate the rollupdate process: `cat <your stanza file> | rollupdate --verbose`If you want to test your input first to make sure it is what you want, and to view the LL reservation job command files and other data files generated by xCAT, run the command with the test option: `cat <your stanza file> | rollupdate -test --verbose`The output files will be placed in the directory you specified in the **jobdir** stanza. The verbose keyword is optional, but allows you to view detailed progress. 

@@ -553,10 +553,10 @@

 ## **Updating xCAT**

-Updating xCAT software as part of the rolling update process is not supported. The xCAT software on your management node can be updated without impacting your running cluster, and should be done manually. See [Setting_Up_a_Linux_xCAT_Mgmt_Node] for instructions on updating your xCAT management node. 
-
-  
-In hierarchical clusters, the xCAT rolling update process should not be used to update the xCAT software on service nodes. See the xCAT hierarchical documents: [Setting_Up_an_AIX_Hierarchical_Cluster] or [Setting_Up_a_Linux_Hierarchical_Cluster]. 
+Updating xCAT software as part of the rolling update process is not supported. The xCAT software on your management node can be updated without impacting your running cluster, and should be done manually. See [Setting_Up_a_Linux_xCAT_Mgmt_Node](Setting_Up_a_Linux_xCAT_Mgmt_Node) for instructions on updating your xCAT management node. 
+
+  
+In hierarchical clusters, the xCAT rolling update process should not be used to update the xCAT software on service nodes. See the xCAT hierarchical documents: [Setting_Up_an_AIX_Hierarchical_Cluster](Setting_Up_an_AIX_Hierarchical_Cluster) or [Setting_Up_a_Linux_Hierarchical_Cluster](Setting_Up_a_Linux_Hierarchical_Cluster). 

 ## **Rolling Updates in a Hierarchical Cluster**

Rolling_Update_Support modified by

Wed, 13 Aug 2014 21:22:36 -0000

--- v74
+++ v75
@@ -272,7 +272,7 @@

 For example, to translate names of the form "bb1s1" to "bb1sn1": translatenames=service:|bb(\d+)s(\d+)|bb($1)sn($2)| 

-To translate names of the form "node20" to "node20-hf2" translatenames=compute:/\z/-hf2/ 
+To translate names of the form "node20" to "node20-hf2" translatenames=compute:/\z/-hf2/ 

 **maxupdates = **_numeric value _| **all**

Rolling_Update_Support modified by

Tue, 12 Aug 2014 21:26:45 -0000

--- v73
+++ v74
@@ -1,4 +1,4 @@
-[[img src=Official-xcat-doc.png]] 
+![](http://sourceforge.net/p/xcat/wiki/XCAT_Documentation/attachment/Official-xcat-doc.png)

 [TOC]

Discussion for Rolling_Update_Support page

Bruce — Mon, 23 Jun 2014 15:32:39 -0000

Rolling_Update_Support modified by Bruce

Bruce — Mon, 23 Jun 2014 15:32:39 -0000

--- v72
+++ v73
@@ -1,3 +1,5 @@
+[[img src=Official-xcat-doc.png]] 
+
 [TOC]

 ## Overview