MRYARN Prototype Version 0.1 Released.
(http://mryarn.sourceforge.net/)
Jul, 20, 2014.
INTRODUCTION
============
Pay-as-you-go is a popular billing model based on users¡¯ resource usage in the cloud. A user¡¯s demand is often changing over time, indicating that it is difficult to keep the high resource utilization all the time for cost efficiency. Resource sharing is an effective approach for high resource utilization. In view of the heterogeneous resource demands of workloads in the cloud, multi-resource allocation fairness is a must for resource sharing in cloud computing. MRYARN is proposed for multi-resource fair allocation on the cloud. It ensures that each user in cloud computing can at least get the amount of total resources as that under the exclusively non-sharing environment in the long term. Moreover, MRYARN can guarantee that no users can get more amount of total allocated resources over time by lying their demands. Finally, MRYARN has a mechanism to discourage users to submit cost-inefficient workloads, especially when there are some idle resources they truly do not need.
Currently, it supports hadoop-2.4.0, and hadoop-2.4.1.
Building
=========
1. Download MRYARN_v_0.1.patch.
2. Download hadoop-2.4.0-src.tar.gz from http://mirror.nus.edu.sg/apache/hadoop/common/hadoop-2.4.0/hadoop-2.4.0-src.tar.gz, and binary hadoop-2.4.0.tar.gz from http://mirror.nus.edu.sg/apache/hadoop/common/hadoop-2.4.0/hadoop-2.4.0.tar.gz
3. Decompress it:
tar xvzf hadoop-2.4.0-src.tar.gz
tar xvzf hadoop-2.4.0.tar.gz
4. cd hadoop-2.4.0-src and run the following command:
patch -p1 < ../MRYARN_v_0.1.patch.
5. build the whole source files in hadoop-2.4.0-src:
mvn clean
mvn package -DskipTests
6. copy the jar files to hadoop-2.4.0
cp -r hadoop-2.4.0-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/target/hadoop-yarn-server-resourcemanager-2.4.0.jar hadoop-2.4.0/share/hadoop/yarn/
cp -r hadoop-2.4.0-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/target/hadoop-yarn-common-2.4.0.jar hadoop-2.4.0/share/hadoop/yarn/
Configuring
===========
1. Edit the file hadoop-2.4.0/etc/hadoop/yarn-site.xml by setting yarn.resourcemanager.scheduler.class with org.apache.hadoop.yarn.server.resourcemanager.scheduler.myfs.FairScheduler, i.e.,
<property>
<name>yarn.resourcemanager.scheduler.class</name>
<value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.myfs.FairScheduler</value>
</property>
2. Configure the hadoop-2.4.0/etc/hadoop/fair-scheduler.xml, e.g.,
<?xml version="1.0"?>
<allocations>
<queue name="Spark">
<schedulingPolicy>fair</schedulingPolicy>
<queueFairShareSchedulingMode>memorable_maxmin_fairshare</queueFairShareSchedulingMode>
</queue>
<queue name="Facebook">
<schedulingPolicy>fair</schedulingPolicy>
<queueFairShareSchedulingMode>memorable_maxmin_fairshare</queueFairShareSchedulingMode>
</queue>
<queue name="Purdue">
<schedulingPolicy>fair</schedulingPolicy>
<queueFairShareSchedulingMode>memorable_maxmin_fairshare</queueFairShareSchedulingMode>
</queue>
<queue name="TPC-H">
<schedulingPolicy>fair</schedulingPolicy>
<queueFairShareSchedulingMode>memorable_maxmin_fairshare</queueFairShareSchedulingMode>
</queue>
<defaultQueueFairShareSchedulingMode>memorable_maxmin_fairshare</defaultQueueFairShareSchedulingMode> <!--memoryless_maxmin_fairshare, memorable_maxmin_fairshare -->
<defaultQueueRoundRobinTimeRoundLength>2592000</defaultQueueRoundRobinTimeRoundLength> <!-- 1 months -->
<defaultQueueSchedulingPolicy>fair</defaultQueueSchedulingPolicy>
</allocations>
3. For other hadoop specific configuration and cluster setup, please refer to http://hadoop.apache.org/docs/r2.4.0/hadoop-project-dist/hadoop-common/ClusterSetup.html.
COPYRIGHT
=========
This software is developed by Tang Shanjiang, School of Computer Engineering,
Nanyang Technological University. If any comments or problems, please directly contact Tang shanjiang
using either of the following email addresses:
stang5@e.ntu.edu.sg; tashj@tju.edu.cn; tashj@sina.com.
MRYARN is an open-source software, complying with Apache License, Version 2.0.
MRYARN is distributed WITHOUT WARRANTY, express or implied. The authors accept NO LEGAL
LIABILITY OR RESPONSIBILITY for loss due to reliance on the program.
To use this software, please cite the following paper:
/******************************************************
Shanjiang Tang, Zhaojie, Niu, Bu-Sung Lee, Bingsheng He:
"Multi-Resource Fair Allocation in Pay-as-you-go Cloud Computing", Technical Report-07-2014, 2014.
*******************************************************/
