Recent changes to AttachingEBS

AttachingEBS modified by Daniel Povey

Daniel Povey — Sat, 14 Dec 2013 03:59:53 -0000

--- v4
+++ v5
@@ -10,7 +10,7 @@

 We will be making these disks available by attaching each one to a particular machine-- not necessarily the master-- and exporting them to the other machines via NFS.  Be careful what you use these EBS volumes for though-- Amazon will not give you fast access to "vanilla" EBS volumes except in short-term bursts.  If you want faster there are two options: either

-  * use use the "iops-provisioned" EBS volumes which are a bit more expensive (add e.g. `-t io1 -i 1000` to the command above), or
+  * Use the "iops-provisioned" EBS volumes which are a bit more expensive (add e.g. `-t io1 -i 1000` to the command above), or
   * Use the "ephemeral storage" of the nodes, which our setup makes available by default as /export/.

 If you use the second option, be careful as it will be lost when you stop or terminate the node, or if it crashes.  You will have to copy it to somewhere else before you shut the node down, if you want to keep it permanently.

WikiPage AttachingEBS modified by Daniel Povey

Daniel Povey — Mon, 25 Mar 2013 18:40:06 -0000

--- v3
+++ v4
@@ -8,37 +8,52 @@
     # ec2tag vol-995797ea  --tag Name=mycluster_home1
     TAG volume  vol-995797ea    Name    mycluster_home1

-We will be making these disks available by attaching each one to a particular machine-- not necessarily the master-- and exporting them to the other machines via NFS.  Be careful what you use these EBS volumes for though-- Amazon will not give you fast access to "vanilla" EBS volumes except in short-term bursts.  If you want faster access you should either use use the "iops-provisioned" EBS volumes which are a bit more expensive, or use the "ephemeral storage" of the nodes, which our setup makes available by default as /export/.
+We will be making these disks available by attaching each one to a particular machine-- not necessarily the master-- and exporting them to the other machines via NFS.  Be careful what you use these EBS volumes for though-- Amazon will not give you fast access to "vanilla" EBS volumes except in short-term bursts.  If you want faster there are two options: either

+  * use use the "iops-provisioned" EBS volumes which are a bit more expensive (add e.g. `-t io1 -i 1000` to the command above), or
+  * Use the "ephemeral storage" of the nodes, which our setup makes available by default as /export/.

-It can be attached to a machine-- not necessarily the master, in fact it is not a good idea to have all volumes attached to the same machine, and I prefer to leave the master without NFS volumes so its network does not get overloaded.  We'll attach this one to the master, just for demonstration: from your local machine,
+If you use the second option, be careful as it will be lost when you stop or terminate the node, or if it crashes.  You will have to copy it to somewhere else before you shut the node down, if you want to keep it permanently.
+
+It can be attached to a machine-- not necessarily the master, in fact it is not a good idea to have all volumes attached to the same machine, and I prefer to leave the master without NFS volumes so its network does not get overloaded.  We'll attach this one to the node `m1-01`.  From your local machine, do:

     :::text
-    ec2-attach-volume vol-995797ea -i i-41401821 -d /dev/xvdba
+    ec2-attach-volume vol-995797ea -i i-7705cd17 -d /dev/xvdz

-You can tell whether it attached correctly by doing `ec2dvol vol-995797ea`; it should say `attached`.  Since the volume is newly created, you will have to format it.  **Caution: only do this if the volume contains no data.  Double-check the device name to avoid formatting an existing volume.**  From the instance, do:
+You can tell whether it attached correctly by doing `ec2dvol vol-995797ea`; it should say `attached`.  Since the volume is newly created, you will have to format it as a file system.  From m1-01, do:

     :::text
-    mkfs -t ext3 /dev/xvdba
+    device=/dev/xvdz
+    cmp <(head -c 1000 $device)  <(head -c 1000 /dev/zero) && mkfs -t ext3 $device

-You can choose a name where you will mount the volume in /mnt, and mount it:
+This command is the same as `mkfs -t ext3 /dev/xvdz`; the part with `cmp` is intended to make it harder for you to accidentally format an already formatted disk.  If you attach more than one EBS volume to the same machine, you will have to use a different device name: choose `/dev/xvdy`, for example.
+
+You can now mount the volume in `/mnt`: from m1-01, do:

     :::text
     mkdir /mnt/home1
-    mount /dev/xvdba /mnt/home1
+    mount /dev/xvdz /mnt/home1
     chmod a+rwx /mnt/home1

-We'll want to export this via NFS, so do as follows:
+You can check that it worked as follows: `df` should tell you it is device `/dev/xvdz`.
+
+    :::text
+    root@m1-01:~# df /mnt/home1
+    Filesystem           1K-blocks      Used Available Use% Mounted on
+    /dev/xvdz             10321208    154232   9642688   2% /mnt/home1
+
+
+We'll want to export this via NFS, so do as follows on m1-01, do:

     :::text
     echo '/mnt/home1      *(rw,sync,no_root_squash)' >> /etc/exports
     service nfs-kernel-server reload

-The command `service nfs-kernel-server reload` will print some harmless warnings about `'subtree_check'`.  If we wanted this directory to be automounted as /exports/home1, we could so as follows:
+The command `service nfs-kernel-server reload` will print some harmless warnings about `'subtree_check'`.  If we wanted this directory to be automounted as /exports/home1, we could so as follows: on `master`,

     :::text
     cd /var/yp
-    echo "home1 master:/mnt/home1" >> ypfiles/auto.export
+    echo "home1 m1-01:/mnt/home1" >> ypfiles/auto.export
     make

 If you did this, to verify that it worked you can do `ls /export/home1`:

WikiPage AttachingEBS modified by Daniel Povey

Daniel Povey — Mon, 25 Mar 2013 05:08:00 -0000

--- v2
+++ v3
@@ -2,6 +2,7 @@

 In order to have relatively permanent storage attached to our cluster, we will use Amazon's "Elastic Block Store"-- these are basically virtual disks that you can attach to nodes.  You can create these disks in any size you specify.  Our experience for speech recognition applications is that the cost of CPU is orders of magnitude more than the cost of storage, so don't obsess too much over the size.  I usually allocate them with a size of 100GB.  We can give them an arbitrary name but try to make it unique:

+    :::text
     # ec2addvol -s 100 -z us-east-1c
     VOLUME  vol-995797ea    100     us-east-1c  creating    2013-03-23T06:10:10+0000    standard
     # ec2tag vol-995797ea  --tag Name=mycluster_home1
@@ -12,36 +13,42 @@

 It can be attached to a machine-- not necessarily the master, in fact it is not a good idea to have all volumes attached to the same machine, and I prefer to leave the master without NFS volumes so its network does not get overloaded.  We'll attach this one to the master, just for demonstration: from your local machine,

+    :::text
     ec2-attach-volume vol-995797ea -i i-41401821 -d /dev/xvdba

 You can tell whether it attached correctly by doing `ec2dvol vol-995797ea`; it should say `attached`.  Since the volume is newly created, you will have to format it.  **Caution: only do this if the volume contains no data.  Double-check the device name to avoid formatting an existing volume.**  From the instance, do:

+    :::text
     mkfs -t ext3 /dev/xvdba

 You can choose a name where you will mount the volume in /mnt, and mount it:

+    :::text
     mkdir /mnt/home1
     mount /dev/xvdba /mnt/home1
     chmod a+rwx /mnt/home1

 We'll want to export this via NFS, so do as follows:

+    :::text
     echo '/mnt/home1      *(rw,sync,no_root_squash)' >> /etc/exports
     service nfs-kernel-server reload

 The command `service nfs-kernel-server reload` will print some harmless warnings about `'subtree_check'`.  If we wanted this directory to be automounted as /exports/home1, we could so as follows:

+    :::text
     cd /var/yp
     echo "home1 master:/mnt/home1" >> ypfiles/auto.export
     make

 If you did this, to verify that it worked you can do `ls /export/home1`:

+    :::text
     # ls /export/home1
     lost+found

 Later we'll show you an easier way to do this using a kluster script.

+Previous: [Adding Nodes](AddingNodes)
 Next: [Adding a user](AddingUser)
 Up: [Kluster Wiki](Home)
-

WikiPage AttachingEBS modified by Daniel Povey

Daniel Povey — Sun, 24 Mar 2013 00:16:26 -0000

--- v1
+++ v2
@@ -1,4 +1,4 @@
-# Attaching EBS storage to nodes.
+## Attaching EBS storage to nodes

 In order to have relatively permanent storage attached to our cluster, we will use Amazon's "Elastic Block Store"-- these are basically virtual disks that you can attach to nodes.  You can create these disks in any size you specify.  Our experience for speech recognition applications is that the cost of CPU is orders of magnitude more than the cost of storage, so don't obsess too much over the size.  I usually allocate them with a size of 100GB.  We can give them an arbitrary name but try to make it unique:

WikiPage AttachingEBS modified by Daniel Povey

Daniel Povey — Sat, 23 Mar 2013 19:45:12 -0000

Attaching EBS storage to nodes.

In order to have relatively permanent storage attached to our cluster, we will use Amazon's "Elastic Block Store"-- these are basically virtual disks that you can attach to nodes. You can create these disks in any size you specify. Our experience for speech recognition applications is that the cost of CPU is orders of magnitude more than the cost of storage, so don't obsess too much over the size. I usually allocate them with a size of 100GB. We can give them an arbitrary name but try to make it unique:

# ec2addvol -s 100 -z us-east-1c
VOLUME  vol-995797ea    100     us-east-1c  creating    2013-03-23T06:10:10+0000    standard
# ec2tag vol-995797ea  --tag Name=mycluster_home1
TAG volume  vol-995797ea    Name    mycluster_home1

We will be making these disks available by attaching each one to a particular machine-- not necessarily the master-- and exporting them to the other machines via NFS. Be careful what you use these EBS volumes for though-- Amazon will not give you fast access to "vanilla" EBS volumes except in short-term bursts. If you want faster access you should either use use the "iops-provisioned" EBS volumes which are a bit more expensive, or use the "ephemeral storage" of the nodes, which our setup makes available by default as /export/.

It can be attached to a machine-- not necessarily the master, in fact it is not a good idea to have all volumes attached to the same machine, and I prefer to leave the master without NFS volumes so its network does not get overloaded. We'll attach this one to the master, just for demonstration: from your local machine,

ec2-attach-volume vol-995797ea -i i-41401821 -d /dev/xvdba

You can tell whether it attached correctly by doing ec2dvol vol-995797ea; it should say attached. Since the volume is newly created, you will have to format it. Caution: only do this if the volume contains no data. Double-check the device name to avoid formatting an existing volume. From the instance, do:

mkfs -t ext3 /dev/xvdba

You can choose a name where you will mount the volume in /mnt, and mount it:

mkdir /mnt/home1
mount /dev/xvdba /mnt/home1
chmod a+rwx /mnt/home1

We'll want to export this via NFS, so do as follows:

echo '/mnt/home1      *(rw,sync,no_root_squash)' >> /etc/exports
service nfs-kernel-server reload

The command service nfs-kernel-server reload will print some harmless warnings about 'subtree_check'. If we wanted this directory to be automounted as /exports/home1, we could so as follows:

cd /var/yp
echo "home1 master:/mnt/home1" >> ypfiles/auto.export
make

If you did this, to verify that it worked you can do ls /export/home1:

# ls /export/home1
lost+found

Later we'll show you an easier way to do this using a kluster script.

Next: Adding a user
Up: Kluster Wiki