Recent changes to AddingNodes

AddingNodes modified by Carl Pupa

Carl Pupa — Fri, 05 Jun 2015 05:46:39 -0000

--- v5
+++ v6
@@ -46,9 +46,9 @@
     :::text
     root@c3-01:~# ls /export/master
     lost+found
-    root@c3-01:~# ls /export/m1-01
+    root@c3-01:~# ls /export/c3-01
     lost+found
-    root@c3-01:~# ls /export/m1-02
+    root@c3-01:~# ls /export/c3-02
     lost+found
     root@c3-01:~# qhost -q
     HOSTNAME                ARCH         NCPU  LOAD  MEMTOT  MEMUSE  SWAPTO  SWAPUS

AddingNodes modified by Carl Pupa

Carl Pupa — Fri, 05 Jun 2015 05:46:06 -0000

--- v4
+++ v5
@@ -54,11 +54,11 @@
     HOSTNAME                ARCH         NCPU  LOAD  MEMTOT  MEMUSE  SWAPTO  SWAPUS
     -------------------------------------------------------------------------------
     global                  -               -     -       -       -       -       -
-    c3-01                   lx26-amd64      2  0.11    7.5G  267.8M    2.9G     0.0
+    c3-01                   lx26-amd64      2  0.11   3.75G  267.8M    1.4G     0.0
        all.q                BIP   0/0/2         
-    c3-02                   lx26-amd64      2  0.54    7.5G  262.3M    2.9G     0.0
+    c3-02                   lx26-amd64      2  0.54   3.75G  262.3M    1.4G     0.0
        all.q                BIP   0/0/2         
-    master                  lx26-amd64      2  0.00    7.5G  267.1M    2.9G     0.0
+    master                  lx26-amd64      2  0.00   3.75G  267.1M    1.4G     0.0
        all.q                BIP   0/0/1

 We tested this a few minutes after launching the nodes.  These export directories are the last thing to come up, because the local disk has to be formatted first and this takes some time.

AddingNodes modified by Carl Pupa

Carl Pupa — Fri, 05 Jun 2015 05:44:23 -0000

--- v3
+++ v4
@@ -3,14 +3,14 @@
 If you want to add a node to your kluster you can do so as follows.  Assuming you have sourced config.sh and vars.sh, you can do something like the following: from your local machine, do

     :::text
-    kl-add-nodes m1.large m1-01 m1-02
+    kl-add-nodes c3.large c3-01 c3-02

-This will add the nodes for you.  In order for you to understand what is really going on here, we will explain what this script does for each node.  Say for the node m1-01, it does something like the following.  First it creates a file `foo` that will become the user-data that will be made available to the instance:
+This will add the nodes for you.  In order for you to understand what is really going on here, we will explain what this script does for each node.  Say for the node c3-01, it does something like the following.  First it creates a file `foo` that will become the user-data that will be made available to the instance:

     :::text
     ## Don't run this! It's part of an explanation ##
     cat >foo <<EOF -="" #hostname="" m1-01="" +="" #hostname="" c3-01="" #hosts_entry="" 10.206.106.228="" master="" EOF="" @@="" -18,45="" +18,45="" @@="" :::text="" ##="" Don't="" run="" this!="" It's="" part="" of="" an="" explanation="" ##="" -="" ec2run="" ami-b4158ddd="" -g="" testcluster="" -k="" testcluster="" -z="" us-east-1c="" -t="" m1.large="" --user-data-file="" foo="" +="" ec2run="" ami-b4158ddd="" -g="" testcluster="" -k="" testcluster="" -z="" us-east-1c="" -t="" c3.large="" --user-data-file="" foo="" This="" spawns="" the="" node,="" and="" the="" init="" scripts="" on="" the="" node="" do="" most="" of="" the="" rest="" of="" the="" configuration.="" The="" script="" `kl-add-nodes`="" now="" has="" to="" tag="" the="" new="" node,="" which="" is="" now="" the="" kluster="" scripts="" on="" your="" local="" machine="" know="" which="" cluster="" it="" belongs="" to.="" The="" script="" does="" something="" like:="" :::text="" ##="" Don't="" run="" this!="" It's="" part="" of="" an="" explanation="" ##="" -="" ec2tag="" i-947f1afa="" --tag="" Name="testcluster-m1-01" +="" ec2tag="" i-947f1afa="" --tag="" Name="testcluster-c3-01" OK,="" explanation="" over.="" Next,="" from="" our="" local="" machine,="" let's="" list="" the="" nodes="" in="" the="" cluster:="" :::text="" #="" kl-list-nodes="" -="" master="" m1.large="" i-50b80a31="" -="" m1-01="" m1.large="" i-6d05cd0d="" -="" m1-02="" m1.large="" i-7705cd17="" +="" master="" c3.large="" i-50b80a31="" +="" c3-01="" c3.large="" i-6d05cd0d="" +="" c3-02="" c3.large="" i-7705cd17="" We="" can="" ssh="" to="" one="" of="" the="" nodes="" (as="" root)="" as="" follows:="" from="" the="" local="" machine,="" :::text="" -="" kl-sshnode="" m1-01="" +="" kl-sshnode="" c3-01="" The="" reader="" familiar="" with="" <a="" href="http://star.mit.edu/cluster/">MIT StarCluster might notice a similarity with some of the command names in that software.  We were inspired by that software, but chose to design our scripts in a simpler way that is intended to be easier to modify.

 Once you are on node m1-01, you can do a couple of things to check whether things are up:

     :::text
-    root@m1-01:~# ls /export/master
+    root@c3-01:~# ls /export/master
     lost+found
-    root@m1-01:~# ls /export/m1-01
+    root@c3-01:~# ls /export/m1-01
     lost+found
-    root@m1-01:~# ls /export/m1-02
+    root@c3-01:~# ls /export/m1-02
     lost+found
-    root@m1-01:~# qhost -q
+    root@c3-01:~# qhost -q
     HOSTNAME                ARCH         NCPU  LOAD  MEMTOT  MEMUSE  SWAPTO  SWAPUS
     -------------------------------------------------------------------------------
     global                  -               -     -       -       -       -       -
-    m1-01                   lx26-amd64      2  0.11    7.5G  267.8M    2.9G     0.0
+    c3-01                   lx26-amd64      2  0.11    7.5G  267.8M    2.9G     0.0
        all.q                BIP   0/0/2         
-    m1-02                   lx26-amd64      2  0.54    7.5G  262.3M    2.9G     0.0
+    c3-02                   lx26-amd64      2  0.54    7.5G  262.3M    2.9G     0.0
        all.q                BIP   0/0/2         
     master                  lx26-amd64      2  0.00    7.5G  267.1M    2.9G     0.0
        all.q                BIP   0/0/1

WikiPage AddingNodes modified by Daniel Povey

Daniel Povey — Mon, 25 Mar 2013 05:02:13 -0000

--- v2
+++ v3
@@ -1,11 +1,14 @@
 ## Adding nodes to your cluster

-If you want to add a node to your kluster you can do so as follows.  Assuming you have sourced config.sh and vars.sh, you can do something like the following:
+If you want to add a node to your kluster you can do so as follows.  Assuming you have sourced config.sh and vars.sh, you can do something like the following: from your local machine, do

+    :::text
     kl-add-nodes m1.large m1-01 m1-02

 This will add the nodes for you.  In order for you to understand what is really going on here, we will explain what this script does for each node.  Say for the node m1-01, it does something like the following.  First it creates a file `foo` that will become the user-data that will be made available to the instance:

+    :::text
+    ## Don't run this! It's part of an explanation ##
     cat >foo <MIT StarCluster might notice a similarity with some of the command names in that software.  We were inspired by that software, but chose to design our scripts in a simpler way that is intended to be easier to modify.
+
+Once you are on node m1-01, you can do a couple of things to check whether things are up:
+
+    :::text
+    root@m1-01:~# ls /export/master
+    lost+found
+    root@m1-01:~# ls /export/m1-01
+    lost+found
+    root@m1-01:~# ls /export/m1-02
+    lost+found
+    root@m1-01:~# qhost -q
+    HOSTNAME                ARCH         NCPU  LOAD  MEMTOT  MEMUSE  SWAPTO  SWAPUS
+    -------------------------------------------------------------------------------
+    global                  -               -     -       -       -       -       -
+    m1-01                   lx26-amd64      2  0.11    7.5G  267.8M    2.9G     0.0
+       all.q                BIP   0/0/2         
+    m1-02                   lx26-amd64      2  0.54    7.5G  262.3M    2.9G     0.0
+       all.q                BIP   0/0/2         
+    master                  lx26-amd64      2  0.00    7.5G  267.1M    2.9G     0.0
+       all.q                BIP   0/0/1         
+
+We tested this a few minutes after launching the nodes.  These export directories are the last thing to come up, because the local disk has to be formatted first and this takes some time.
+
+Previous: [Setting up your Kluster config](SettingConfig)
+Next: [Adding a user](AddingUser)
+Up: [Kluster Wiki](Home)
+

WikiPage AddingNodes modified by Daniel Povey

Daniel Povey — Mon, 25 Mar 2013 00:21:16 -0000

WikiPage AddingNodes modified by Daniel Povey

Daniel Povey — Sun, 24 Mar 2013 23:06:24 -0000

Adding nodes to your cluster

If you want to add a node to your kluster you can do so as follows. Assuming you have sourced config.sh and vars.sh, you can do something like the following:

kl-add-nodes m1.large m1-01 m1-02

This will add the nodes for you. In order for you to understand what is really going on here, we will explain what this script does for each node. Say for the node m1-01, it does something like the following. First it creates a file foo that will become the user-data that will be made available to the instance:

cat >foo <<EOF                                                                                                                        
#hostname m1-01
#hosts_entry 10.206.106.228 master                                                                                                        
EOF

This file will be parsed by the init-script /etc/init.d/kluster-set-hostname. If node gets user-data like this it becomes a node; by default it becomes the master. Next, kl-add-nodes will run a command like the following:

ec2run ami-b4158ddd -g testcluster -k testcluster -z us-east-1c -t m1.large --user-data-file foo

This spawns the node, and the init scripts on the node do most of the rest of the configuration. The script kl-add-nodes now has to tag the new node, which is now the kluster scripts on your local machine know which cluster it belongs to:

ec2tag i-947f1afa --tag Name=testcluster-m1-01