http://www.tutorialspoint.com/hadoop/hadoop_hdfs_overview.htm

HADOOP - HDFS OVERVIEW

Hadoop File System was developed using distributed file system design. It is run on commodity

hardware. Unlike other distributed systems, HDFS is highly faulttolerant and designed using low-

cost hardware.

HDFS holds very large amount of data and provides easier access. To store such huge data, the

files are stored across multiple machines. These files are stored in redundant fashion to rescue the

system from possible data losses in case of failure. HDFS also makes applications available to

parallel processing.

Features of HDFS

It is suitable for the distributed storage and processing.

Hadoop provides a command interface to interact with HDFS.

The built-in servers of namenode and datanode help users to easily check the status of

cluster.

Streaming access to file system data.

HDFS provides file permissions and authentication.

HDFS Architecture

Given below is the architecture of a Hadoop File System.

HDFS follows the master-slave architecture and it has the following elements.

Namenode

The namenode is the commodity hardware that contains the GNU/Linux operating system and the

namenode software. It is a software that can be run on commodity hardware. The system having

the namenode acts as the master server and it does the following tasks:

Manages the file system namespace.

Regulates client’s access to files.

It also executes file system operations such as renaming, closing, and opening files and

directories.

Datanode

The datanode is a commodity hardware having the GNU/Linux operating system and datanode

software. For every node Commodityhardware/System in a cluster, there will be a datanode. These nodes

manage the data storage of their system.

Datanodes perform read-write operations on the file systems, as per client request.

They also perform operations such as block creation, deletion, and replication according to

the instructions of the namenode.

Block

Generally the user data is stored in the files of HDFS. The file in a file system will be divided into

one or more segments and/or stored in individual data nodes. These file segments are called as

blocks. In other words, the minimum amount of data that HDFS can read or write is called a Block.

The default block size is 64MB, but it can be increased as per the need to change in HDFS

configuration.

Goals of HDFS

Fault detection and recovery : Since HDFS includes a large number of commodity

hardware, failure of components is frequent. Therefore HDFS should have mechanisms for

quick and automatic fault detection and recovery.

Huge datasets : HDFS should have hundreds of nodes per cluster to manage the

applications having huge datasets.

Hardware at data : A requested task can be done efficiently, when the computation takes

place near the data. Especially where huge datasets are involved, it reduces the network

traffic and increases the throughput.

Loading [MathJax]/jax/output/HTML-CSS/fonts/TeX/fontdata.js

Hadoop - HDFS Overview

Hadoop - HDFS Overview