Menu
▾
▴
Home
Hadoop Compatible Framework For Discovering Network Topology And Detecting Hardware Failures
Abstract
The project involves detection of hardware failures and discovery of the network topology within the Hadoop cluster. The application developed, collects various network and hardware components information from all the DataNodes and analyses them to detect the network, CPU and hard disk failures. It guarantees recent, periodic and automated collection of information from all the DataNodes. The data collected is stored in the Hadoop Distributed File System (HDFS), which is a primary shared storage system. The analysis of data is performed using Apache Pig tool, which is a platform for analyzing large data sets. The application checks for any failures in a DataNode by comparing it with the threshold values of failure and intimates the failed node about the failure so that it can take appropriate actions. The project also discovers the network topology of all the DataNodes in the Hadoop cluster with the help of a network monitoring tool. The application can figure out whether a host is up or down, status of ports, the operating system of the target, estimated uptime and presence of a firewall. The network topology discovered through our project can be used to perform load balancing in the Hadoop network.
Project Members
Manepalli V Prabha Satya
Raja Vidya Varsha
Sheeba Samuel
Shwetha Muralidharan
Anusha Ventrapragada.
Technical Report
http://www.iiitb.ac.in/sites/default/files/uploads/IIITB-TR-2012-005.pdf