RainforestCluster

RainforestCluster is an Amazon EC2 python program that manages and load-balances dynamic clusters to allow for maximum workflow flexibility and speed at minimal cost. It enables one to quickly and cheaply create dynamic compute clusters in the cloud, which can then run computational pipelines generically. It is also able to optimize the use of spot instances - idle computers in Amazon's cloud that are available at drastically reduced cost (5x-10x cheaper) - but can be terminated at any moment if capacity drops or the bid price rises. It also provides pre-installed features such as GlusterFS distributed filesystems, ThunderstormDistributor queuing system, RAID 0 /scratch, password-less ssh, and automatic cluster management, for ease of use and maximum processing speed for computational tools. Originally it was developed as a different version for the Wall Lab at Harvard CBMI.

Features

Create Amazon EC2 custom clusters dynamically
Add, remove, and loadbalance nodes
Create large shared gluster volumes with spanning across multiple masters
Distribute jobs efficiently using ThunderstormDistributor
View machine, job, and cluster statistics on a web-based interface

Project Activity

See All Activity >

License

MIT License

Follow RainforestCluster

RainforestCluster Web Site

Other Useful Business Software

Gen AI apps are built with MongoDB Atlas

The database for AI-powered applications.

MongoDB Atlas is the developer-friendly database used to build, scale, and run gen AI and LLM-powered apps—without needing a separate vector database. Atlas offers built-in vector search, global availability across 115+ regions, and flexible document modeling. Start building AI apps faster, all in one place.

Start Free

Rate This Project

User Reviews

Be the first to post a review of RainforestCluster!

Additional Project Details

Operating Systems

Cygwin, Linux, Mac

Intended Audience

Information Technology, Science/Research

User Interface

Console/Terminal

Programming Language

Python

Related Categories

Python Systems Administration Software, Python Clustering Software, Python Distributed Computing Software

Registered

2014-03-09

Similar Business Software

AWS Batch

AWS Batch enables developers, scientists, and engineers to easily and efficiently run hundreds of thousands of batch computing jobs on AWS. AWS Batch dynamically provisions the optimal quantity and type of compute resources (e.g., CPU or memory optimized instances) based on the volume and...

See Software
EC2 Spot

Amazon EC2 Spot Instances let you take advantage of unused EC2 capacity in the AWS cloud. Spot Instances are available at up to a 90% discount compared to On-Demand prices. You can use Spot Instances for various stateless, fault-tolerant, or flexible applications such as big data, containerized...

See Software
AWS Elastic Fabric Adapter (EFA)

Elastic Fabric Adapter (EFA) is a network interface for Amazon EC2 instances that enables customers to run applications requiring high levels of inter-node communications at scale on AWS. Its custom-built operating system (OS) bypass hardware interface enhances the performance of inter-instance...

See Software