File	Date	Author	Commit
rainforestcluster	2014-03-10	David Tulga	[0a2e89] RainforestCluster 0.9 Beta
AMI_INSTALL	2014-03-10	David Tulga	[0a2e89] RainforestCluster 0.9 Beta
INSTALL	2014-03-10	David Tulga	[0a2e89] RainforestCluster 0.9 Beta
LICENSE	2014-03-10	David Tulga	[0a2e89] RainforestCluster 0.9 Beta
README	2014-03-10	David Tulga	[0a2e89] RainforestCluster 0.9 Beta
rfcluster_remote.py	2014-03-10	David Tulga	[0a2e89] RainforestCluster 0.9 Beta

Read Me

RainforestCluster 0.9 Beta (2014-3-10)
by David Tulga
Originally created for the Wall Lab at Harvard CBMI

See INSTALL for install instructions
See AMI_INSTALL for AMI creation and customization instructions
See LICENSE for license text (Open Source MIT-style license)


=== Description ===

This is an Amazon EC2 python program that manages and load-balances dynamic clusters to allow for maximum workflow flexibility, minimizing costs and maximizing workflow speed. It enables one to quickly and cheaply create dynamic compute clusters in the cloud, which can then run Cosmos, GenomeKey, or other computational pipelines generically. It is also able to optimize for the use of Spot Instances - idle computers in Amazon's cloud that are available at drastically reduced cost (5x-10x cheaper) - but can be terminated at any moment if capacity drops or the bid price rises.  It also provides pre-installed features such as GlusterFS distributed filesystems, RAID0 /scratch, passwordless ssh, and automatic cluster connection, for ease of use and maximum processing speed for the computational tools. 


=== Usage ===

rainforestcluster <global_command> <command_args>
rfcluster <global_command> <command_args>

rainforestcluster <cluster_command> <cluster_name> <command_args>
rfcluster <cluster_command> <cluster_name> <command_args>


== Global Commands ==
(All commands can be as command or --command)

help : Print general help

version : Print Current Version

list listclusters : Lists all the clusters manageable by rainforestcluster
	-l --long : Lists extra information about each cluster, such as state, size, etc.

== Cluster Commands ==
(All commands must be preceded with the cluster name, and can also be as command or --command)

start <cluster_parameters> : start a cluster
	--cluster -c --template : Specify a cluster template other than the default
	--loadbalance <min> <max>
	--loadbalance <max> (where min = config default or 0 if not set)
	--loadbalance (will use the config defaults) : Will start a loadbalancer loop to change the cluster dynamically based on load, the min and max parameters are the number of nodes (0 means only the master is active)

status : get cluster status

sshmaster <ssh options> : ssh to the master instance for the cluster (ALSO: sshsuper)

sshto <machine> (<ssh_options>) : ssh to a given machine name in the cluster (ALSO: ssh)

put <local file> (<optional: remote location>) : scp a local file to the cluster, defaults to ... (master instance only)

get <remote file> (<optional: local location>) : scp a remote file to the local computer, defaults to the current directory (master instance only)

addnodes <number of nodes> : Add the specified number of nodes to the cluster

removenodes <number of nodes> [OR: <list of node names separated by a space>]: Remove either the specified number of nodes (chosen by least active first), or the nodes given by name (same as terminate --machine <...>)

loadbalance <min> <max> : loadbalance <max> : loadbalance : Will start a loadbalancer loop to change the cluster size as needed due to job load (can also be run on Amazon???)

terminate : fully terminate a cluster, removing additional security groups, spot requests, instances, etc. (will not delete any ebs volumes (except instance boot volumes) or S3 data.)
	--nodes_only : Only terminate all nodes
	--node --machine -m <machine_name(s)> : or : <machine name(s)> : Only terminate the given machine(s)

Simultaneous/thread safety: Help and version at any time, sshmaster, put, and get are safe to run simulataneously once a cluster is fully active,
start and terminate can only run by themselves with no other commands at the same time, and add/remove/loadbalance allow other commands, but only one in this group at a time.


Examples:
rainforestcluster help
rfcluster help

rfcluster start test
rfcluster status test
rfcluster sshmaster test
rfcluster sshmaster test -YC -L 8080:localhost:8080
rfcluster put test /home/user/file1.txt /home/ubuntu/
rfcluster put test file2.txt  								(Defaults to placing it in either the custom user or ubuntu's home directory)
rfcluster get test /gluster/gv0/file3.txt /home/user/Downloads/
rfcluster get test /home/ubuntu/file4.txt 					(Defaults to placing it the current directory)
rfcluster get test "~/file4.txt" 							(Remote ~ must be in quotes)
rfcluster addnodes test 2
rfcluster removenodes test 2
rfcluster removenodes test node005 node006
rfcluster loadbalance test
rfcluster loadbalance test 0 10
rfcluster terminate test