MapReduce

Distributed computing

can get very complicated

Managing resources and

memory across multiple nodes

MapReduce

Distributed computing

can get very complicated

What to do if a node

goes down?

MapReduce

MapReduce abstracts

the programmer from

all these complications

MapReduce

You just deﬁne 2

functions

map() reduce()

MapReduce

map() reduce()

The rest is taken care

of by Hadoop!

Let’s take an example

We have a large text ﬁle

Objective: Create a

Frequency

Distribution of

words in the ﬁle

We have a large text ﬁle

This is a pretty

common task in

Natural Language

Processing

We have a large text ﬁle

Word

Count

because

each

ﬁgure

The text ﬁle has been divided

into blocks and stored in HDFS

Data node 1

Data node 2

Block 1

Block 2

Data node 3

Block 3

Name node

The name

node stores

metadata

Here is how the data ﬂows in a

MapReduce job

Data node 1

Hey Diddle Diddle

Block 1

…

Data node 2

The cat and the ﬁddle

Block 2

….

Data node 3

The cow jumped over the moon

Block 3

…

Each block here would represent a

part of the text ﬁle

Here is a snapshot of some text

from each block

Data node 1

Hey Diddle Diddle

…

Data node 2

The cat and the ﬁddle

….

MAP

Data node 3

The cow jumped over the moon

…

MAP

<Hey,1>

<Diddle,1>

…..

<The,1>

<Cat,1>

…..

<The,1>

<Cow,1>

…..

The map step will

generate a list of key-

value pairs on each node

From now on all the

inputs and outputs are

formatted as

<key,value> pairs

Data node 1

Hey Diddle Diddle

…

Data node 2

The cat and the ﬁddle

….

MAP

Data node 3

The cow jumped over the moon

…

MAP

<Hey,1>

<Diddle,1>

…..

<The,1>

<Cat,1>

…..

<The,1>

<Cow,1>

…..

These are all

copied over to one

single node

On that node an

operation called

Sort/Merge occurs

Data node 1

Hey Diddle Diddle

…

Data node 2

The cat and the ﬁddle

….

MAP

Data node 3

The cow jumped over the moon

…

MAP

Input

<Hey,1>

<Diddle,1>

…..

<The,1>

<Cat,1>

…..

<The,1>

<Cow,1>

…..

Sort/Merge

<And,1>

<Cat,1>

<Cow,1>

<Diddle,(1,1)>

<Fiddle,1>

<Jumped,1>

<Moon,1>

<Over,1>

<The,

(1,1,1,1)>

Reduce

<And,1>

<Cat,1>

<Cow,1>

<Diddle,2>

<Fiddle,1>

<Jumped,1>

<Moon,1>

<Over,1>

<The,4>

Output

The algorithm for these 2 steps need to be deﬁned by the userData node 1

Hey Diddle Diddle

…

Data node 2

The cat and the ﬁddle

….

MAP

Data node 3

The cow jumped over the moon

…

MAP

Input

<Hey,1>

<Diddle,1>

…..

<The,1>

<Cat,1>

…..

<The,1>

<Cow,1>

…..

Sort/Merge

<And,1>

<Cat,1>

<Cow,1>

<Diddle,(1,1)>

<Fiddle,1>

<Jumped,1>

<Moon,1>

<Over,1>

<The,

(1,1,1,1)>

Reduce

<And,1>

<Cat,1>

<Cow,1>

<Diddle,2>

<Fiddle,1>

<Jumped,1>

<Moon,1>

<Over,1>

<The,4>

Output

Data node 1

Hey Diddle Diddle

…

MAP

<Hey,1>

<Diddle,1>

…..

This function will run once

for each line of the text ﬁle

Data node 1

Hey Diddle Diddle

…

MAP

<Hey,1>

<Diddle,1>

…..

Both the input and output need

to be formatted as <key, value>

pairs

Data node 1

<34, Hey Diddle Diddle>

Hey Diddle Diddle

…

MAP

<Hey,1>

<Diddle,1>

…..

<lineNumber, line>

Data node 1

<34, Hey Diddle Diddle>

…

MAP

<Hey,1>

<Diddle,1>

…..

<word, 1>

There will be one such pair for

each word in the line

Data node 1

<34, Hey Diddle Diddle>

…

MAP

<Hey,1>

<Diddle,1>

…..

This operation can run in

parallel on each data node,

there is no interdependency in

the inputs and outputs

Data node 1

Hey Diddle Diddle

…

Data node 2

The cat and the ﬁddle

….

MAP

Data node 3

The cow jumped over the moon

…

MAP

Input

<Hey,1>

<Diddle,1>

…..

<The,1>

<Cat,1>

…..

<The,1>

<Cow,1>

…..

Sort/Merge

<And,1>

<Cat,1>

<Cow,1>

<Diddle,(1,1)>

<Fiddle,1>

<Jumped,1>

<Moon,1>

<Over,1>

<The,

(1,1,1,1)>

Reduce

<And,1>

<Cat,1>

<Cow,1>

<Diddle,2>

<Fiddle,1>

<Jumped,1>

<Moon,1>

<Over,1>

<The,4>

Output

Sort/Merge

At the end of the map phase, we

have a set of key-value pairs

from each data node

All of these results are

ﬁrst copied over to a

single node

<And,1>

<Cat,1>

<Cow,1>

<Diddle,(1,1)>

<Fiddle,1>

<Jumped,1>

<Moon,1>

<Over,1>

<The,

(1,1,1,1)>

<Hey,1>

<Diddle,1>

…..

<The,1>

<Cat,1>

…..

<The,1>

<Cow,1>

…..

Sort/Merge

<Hey,1>

<Diddle,1>

…..

<The,1>

<Cat,1>

…..

<The,1>

<Cow,1>

…..

<Hey,1>

<Diddle,1>

<The,1>

<Cat,1>

<And,1>

<The,1>

<Fiddle,1>

<The,1>

<Cow,1>

<Jumped,1>

<Over,1>

<The,1>

<Moon,1>

The data is

sorted by key

<And,1>

<Cat,1>

<Cow,1>

<Diddle,(1,1)>

<Fiddle,1>

<Jumped,1>

<Moon,1>

<Over,1>

<The,

(1,1,1,1)>

Sort/Merge

<Hey,1>

<Diddle,1>

…..

<The,1>

<Cat,1>

…..

<The,1>

<Cow,1>

…..

<Hey,1>

<Diddle,1>

<The,1>

<Cat,1>

<And,1>

<The,1>

<Fiddle,1>

<The,1>

<Cow,1>

<Jumped,1>

<Over,1>

<The,1>

<Moon,1>

<And,1>

<Cat,1>

<Cow,1>

<Diddle,(1,1)>

<Fiddle,1>

<Jumped,1>

<Moon,1>

<Over,1>

<The,

(1,1,1,1)>

Sort/Merge

<Hey,1>

<Diddle,1>

…..

<The,1>

<Cat,1>

…..

<The,1>

<Cow,1>

…..

<Hey,1>

<Diddle,1>

<The,1>

<Cat,1>

<And,1>

<The,1>

<Fiddle,1>

<The,1>

<Cow,1>

<Jumped,1>

<Over,1>

<The,1>

<Moon,1>

<And,1>

<Cat,1>

<Cow,1>

<Diddle,1>

<Fiddle,1>

<Jumped,1>

<Moon,1>

<Over,1>

<The,1>

<And,1>

<Cat,1>

<Cow,1>

<Diddle,(1,1)>

<Fiddle,1>

<Jumped,1>

<Moon,1>

<Over,1>

<The,

(1,1,1,1)>

<Hey,1>

<Diddle,1>

…..

<The,1>

<Cat,1>

…..

<The,1>

<Cow,1>

…..

Sort/Merge

<And,1>

<Cat,1>

<Cow,1>

<Diddle,1>

<Fiddle,1>

<Jumped,1>

<Moon,1>

<Over,1>

<The,1>

<And,1>

<Cat,1>

<Cow,1>

<Diddle,(1,1)>

<Fiddle,1>

<Jumped,1>

<Moon,1>

<Over,1>

<The,

(1,1,1,1)>

<Hey,1> <Diddle,1> <Diddle,1> <The,1> <Cat,1> <And,1> <The,1> <Fiddle,1> <The,1> <Cow,1> <Jumped,1> <Over,1> <The,1> <Moon,1>Key-Value pairs with the same key are merged <Hey,1>

<Diddle,1>

…..

<The,1>

<Cat,1>

…..

<The,1>

<Cow,1>

…..

Sort/Merge

<And,1>

<Cat,1>

<Cow,1>

<Diddle,1>

<Fiddle,1>

<Jumped,1>

<Moon,1>

<Over,1>

<The,1>

<And,1>

<Cat,1>

<Cow,1>

<Diddle,(1,1)>

<Fiddle,1>

<Jumped,1>

<Moon,1>

<Over,1>

<The,

(1,1,1,1)>

<Hey,1> <Diddle,1> <Diddle,1> <The,1> <Cat,1> <And,1> <The,1> <Fiddle,1> <The,1> <Cow,1> <Jumped,1> <Over,1> <The,1> <Moon,1>Data node 1

Hey Diddle Diddle

…

Data node 2

The cat and the ﬁddle

….

MAP

Data node 3

The cow jumped over the moon

…

MAP

Input

<Hey,1>

<Diddle,1>

…..

<The,1>

<Cat,1>

…..

<The,1>

<Cow,1>

…..

Sort/Merge

<And,1>

<Cat,1>

<Cow,1>

<Diddle,(1,1)>

<Fiddle,1>

<Jumped,1>

<Moon,1>

<Over,1>

<The,

(1,1,1,1)>

Reduce

<And,1>

<Cat,1>

<Cow,1>

<Diddle,2>

<Fiddle,1>

<Jumped,1>

<Moon,1>

<Over,1>

<The,4>

Output

Data node 1

Hey Diddle Diddle

…

Data node 2

The cat and the ﬁddle

….

MAP

Data node 3

The cow jumped over the moon

…

MAP

Input

<Hey,1>

<Diddle,1>

…..

<The,1>

<Cat,1>

…..

<The,1>

<Cow,1>

…..

Sort/Merge

<And,1>

<Cat,1>

<Cow,1>

<Diddle,(1,1)>

<Fiddle,1>

<Jumped,1>

<Moon,1>

<Over,1>

<The,

(1,1,1,1)>

Reduce

<And,1>

<Cat,1>

<Cow,1>

<Diddle,2>

<Fiddle,1>

<Jumped,1>

<Moon,1>

<Over,1>

<The,4>

Output

<And,1>

<Cat,1>

<Cow,1>

<Diddle,(1,1)>

<Fiddle,1>

<Jumped,1>

<Moon,1>

<Over,1>

<The,

(1,1,1,1)>

Reduce

<And,1>

<Cat,1>

<Cow,1>

<Diddle,2>

<Fiddle,1>

<Jumped,1>

<Moon,1>

<Over,1>

<The,4>

reduce() will run on each pair

generated by the Sort/Merge step

<And,1>

<Cat,1>

<Cow,1>

<Diddle,(1,1)>

<Fiddle,1>

<Jumped,1>

<Moon,1>

<Over,1>

<The,

(1,1,1,1)>

Reduce

<And,1>

<Cat,1>

<Cow,1>

<Diddle,2>

<Fiddle,1>

<Jumped,1>

<Moon,1>

<Over,1>

<The,4>

<word, list>

<And,1>

<Cat,1>

<Cow,1>

<Diddle,(1,1)>

<Fiddle,1>

<Jumped,1>

<Moon,1>

<Over,1>

<The,

(1,1,1,1)>

Reduce

<And,1>

<Cat,1>

<Cow,1>

<Diddle,2>

<Fiddle,1>

<Jumped,1>

<Moon,1>

<Over,1>

<The,4>

<word, list> <word, count>

Data node 1

Hey Diddle Diddle

…

Data node 2

The cat and the ﬁddle

….

MAP

Data node 3

The cow jumped over the moon

…

MAP

Input

<Hey,1>

<Diddle,1>

…..

<The,1>

<Cat,1>

…..

<The,1>

<Cow,1>

…..

Sort/Merge

<And,1>

<Cat,1>

<Cow,1>

<Diddle,(1,1)>

<Fiddle,1>

<Jumped,1>

<Moon,1>

<Over,1>

<The,

(1,1,1,1)>

Reduce

<And,1>

<Cat,1>

<Cow,1>

<Diddle,2>

<Fiddle,1>

<Jumped,1>

<Moon,1>

<Over,1>

<The,4>

Output

Data node 1

Hey Diddle Diddle

…

Data node 2

The cat and the ﬁddle

….

MAP

Data node 3

The cow jumped over the moon

…

MAP

Input

<Hey,1>

<Diddle,1>

…..

<The,1>

<Cat,1>

…..

<The,1>

<Cow,1>

…..

Sort/Merge

<And,1>

<Cat,1>

<Cow,1>

<Diddle,(1,1)>

<Fiddle,1>

<Jumped,1>

<Moon,1>

<Over,1>

<The,

(1,1,1,1)>

Reduce

<And,1>

<Cat,1>

<Cow,1>

<Diddle,2>

<Fiddle,1>

<Jumped,1>

<Moon,1>

<Over,1>

<The,4>

Output

The output is sorted!!Data node 1

Hey Diddle Diddle

…

Data node 2

The cat and the ﬁddle

….

MAP

Data node 3

The cow jumped over the moon

…

MAP

Input

<Hey,1>

<Diddle,1>

…..

<The,1>

<Cat,1>

…..

<The,1>

<Cow,1>

…..

Sort/Merge

<And,1>

<Cat,1>

<Cow,1>

<Diddle,(1,1)>

<Fiddle,1>

<Jumped,1>

<Moon,1>

<Over,1>

<The,

(1,1,1,1)>

Reduce

<And,1>

<Cat,1>

<Cow,1>

<Diddle,2>

<Fiddle,1>

<Jumped,1>

<Moon,1>

<Over,1>

<The,4>

Output

This is a happy byproduct of the Sort/Merge operationData node 1

Hey Diddle Diddle

…

Data node 2

The cat and the ﬁddle

….

MAP

Data node 3

The cow jumped over the

moon

MAP

Input

<Hey,1>

<Diddle,1>

…..

<The,1>

<Cat,1>

…..

<The,1>

<Cow,1>

…..

Sort/Merge

<And,1>

<Cat,1>

<Cow,1>

<Diddle,

(1,1)>

<Fiddle,1>

<Jumped,

<Moon,1>

<Over,1>

Reduce

<And,1>

<Cat,1>

<Cow,1>

<Diddle,2>

<Fiddle,1>

<Jumped,

<Moon,1>

<Over,1>

<The,4>

Output

This is the general idea behind every

MapReduce operation

Data node 1

Hey Diddle Diddle

…

Data node 2

The cat and the ﬁddle

….

MAP

Data node 3

The cow jumped over the

moon

MAP

Input

<Hey,1>

<Diddle,1>

…..

<The,1>

<Cat,1>

…..

<The,1>

<Cow,1>

…..

Sort/Merge

<And,1>

<Cat,1>

<Cow,1>

<Diddle,

(1,1)>

<Fiddle,1>

<Jumped,

<Moon,1>

<Over,1>

Reduce

<And,1>

<Cat,1>

<Cow,1>

<Diddle,2>

<Fiddle,1>

<Jumped,

<Moon,1>

<Over,1>

<The,4>

Output

The map function is chosen such that it

can run in parallel on all the nodes

Data node 1

Hey Diddle Diddle

…

Data node 2

The cat and the ﬁddle

….

MAP

Data node 3

The cow jumped over the

moon

MAP

Input

<Hey,1>

<Diddle,1>

…..

<The,1>

<Cat,1>

…..

<The,1>

<Cow,1>

…..

Sort/Merge

<And,1>

<Cat,1>

<Cow,1>

<Diddle,

(1,1)>

<Fiddle,1>

<Jumped,

<Moon,1>

<Over,1>

Reduce

<And,1>

<Cat,1>

<Cow,1>

<Diddle,2>

<Fiddle,1>

<Jumped,

<Moon,1>

<Over,1>

<The,4>

Output

The output of map should be key, value

pairs

Data node 1

Hey Diddle Diddle

…

Data node 2

The cat and the ﬁddle

….

MAP

Data node 3

The cow jumped over the

moon

MAP

Input

<Hey,1>

<Diddle,1>

…..

<The,1>

<Cat,1>

…..

<The,1>

<Cow,1>

…..

Sort/Merge

<And,1>

<Cat,1>

<Cow,1>

<Diddle,

(1,1)>

<Fiddle,1>

<Jumped,

<Moon,1>

<Over,1>

Reduce

<And,1>

<Cat,1>

<Cow,1>

<Diddle,2>

<Fiddle,1>

<Jumped,

<Moon,1>

<Over,1>

<The,4>

Output

The reduce() function will combine all the

values for the same key in some way

MapReduce Hello World

Let’s write our ﬁrst

MapReduce program

MapReduce Programs are

usually written in Java

Hadoop has an API called Streaming

that can be used for Ruby, Python etc

We’ll get to that a little later

Objective: Create a Frequency

Distribution of words in a text ﬁle

Step 1: Write a map() function

Step 2: Write a reduce() function

Step 3: Setup a driver that points to our

map and reduce implementations

Objective: Create a Frequency

Distribution of words in a text ﬁle

Data node 1

Hey Diddle Diddle

…

Data node 2

The cat and the ﬁddle

….

MAP

Data node 3

The cow jumped over the

moon

MAP

Input

<Hey,1>

<Diddle,1>

…..

<The,1>

<Cat,1>

…..

<The,1>

<Cow,1>

…..

Sort/Merge

<And,1>

<Cat,1>

<Cow,1>

<Diddle,

(1,1)>

<Fiddle,1>

<Jumped,

<Moon,1>

<Over,1>

Reduce

<And,1>

<Cat,1>

<Cow,1>

<Diddle,2>

<Fiddle,1>

<Jumped,

<Moon,1>

<Over,1>

<The,4>

Output

WordMapper Class

Mapper Class

The map() function

is implemented in a

class that extends

the Mapper Class

<And,1> <Cat,1> <Cow,1> <Diddle,(1,1)> <Fiddle,1> <Jumped,1> <Moon,1> <Over,1> Sort/MergeReduce<And,1> <Cat,1> <Cow,1> <Diddle,2> <Fiddle,1> <Jumped,1> <Moon,1> <Over,1> <The,4>WordMapper Class

<input key type,

input value type,

output key type,

output value type>

Mapper Class

This is a generic

class, with 4

type parameters

<And,1> <Cat,1> <Cow,1> <Diddle,(1,1)> <Fiddle,1> <Jumped,1> <Moon,1> <Over,1> Sort/MergeReduce<And,1> <Cat,1> <Cow,1> <Diddle,2> <Fiddle,1> <Jumped,1> <Moon,1> <Over,1> <The,4>Sort/Merge

<And,1>

<Cat,1>

<Cow,1>

<Diddle,

(1,1)>

<Fiddle,1>

<Jumped,

<Moon,1>

<Over,1>

Reduce

<And,1>

<Cat,1>

<Cow,1>

<Diddle,2>

<Fiddle,1>

<Jumped,

<Moon,1>

<Over,1>

<The,4>

WordMapper ClassMapper Class<input key type, input value type, output key type, output value type>WordReducer Class

Reducer Class

The reduce() function is

implemented in a class that

extends the Reducer Class

WordMapper ClassMapper Class<input key type, input value type, output key type, output value type>WordReducer Class

<input key type,

input value type,

output key type,

output value type>

Reducer Class

This is a generic class too,

with 4 type parameters

WordMapper ClassMapper Class<input key type, input value type, output key type, output value type>WordMapper Class

<input key type,

input value type,

output key type,

output value type>

Mapper Class

WordReducer Class

<input key type,

input value type,

output key type,

output value type>

Reducer Class

The output types of the

Mapper should match the

input types of the Reducer

WordMapper Class

<input key type,

input value type,

output key type,

output value type>

Mapper Class

WordReducer Class

<input key type,

input value type,

output key type,

output value type>

Reducer Class

The output types of the

Mapper should match the

input types of the Reducer

WordMapper Class

WordReducer Class

These 2 classes are used

by a Job that is conﬁgured

in the Main Class

WordCount Class

Job Object

WordMapper Class

WordReducer Class

The Job has a

bunch of

properties

that need to

be conﬁgured

WordCount Class

Input ﬁlepath

Output ﬁlepath

Job Object

Mapper class

Reducer class

Output data types

WordMapper Class

WordReducer Class

The Mapper and

Reducer will

point to the

classes with our

implementation

WordCount Class

Job Object

Mapper class

Reducer class

Output data typesInput ﬁlepathOutput ﬁlepathWordMapper Class

WordReducer Class

The output data

types need to

match the type

parameters of

the Mapper and

Reducer Classes

WordCount Class

Job Object

Output data types

Mapper classReducer classInput ﬁlepathOutput ﬁlepathAll these classes

are put into a JAR

ﬁle which contains

all the Hadoop

JARS as well

WordMapper Class

WordReducer Class

WordCount Class

Job Object

When you run a job,

this JAR ﬁle gets

distributed to all the

nodes where the

computation is run

WordCount.JAR

WordMapper Class

WordReducer Class

WordCount Class

WordMapper Class

package com.company;

import org.apache.hadoop.io.IntWritable;

import org.apache.hadoop.io.LongWritable;

import org.apache.hadoop.io.Text;

import org.apache.hadoop.mapreduce.Mapper;

import java.io.IOException;

In Java the map() function is

represented by the Mapper Class

public class WordCountMapper extends Mapper<LongWritable,Text,Text,IntWritable> {

@Override

public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException

{

String line = value.toString();

for (String word : line.split(" ")){

if(word.length()>0){

context.write(new Text(word),new IntWritable(1));

}

WordMapper Class

In Java the map() function is

represented by the Mapper Class

This class represents the

code for the step

<1,Hey diddle diddle>

<Linenum, Line of text>

map()

<Hey,1>

<Diddle,1>

<Word, 1>

package  import import import import  import  public class @Override {   String line = value.toString(); context.write( } }}  }WordMapper Class

package com.company;

import org.apache.hadoop.io.IntWritable;

import org.apache.hadoop.io.LongWritable;

import org.apache.hadoop.io.Text;

import org.apache.hadoop.mapreduce.Mapper;

import java.io.IOException;

Mapper class is a generic

It has 4 type parameters

Input

Key Type

Output

key type

public class WordCountMapper extends Mapper<LongWritable,Text,Text,IntWritable>

{

@Override

public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException{

Input

Value Type

Output

Value type

String line = value.toString();

for (String word : line.split(" ")){

if(word.length()>0){

context.write(new Text(word),new IntWritable(1));

}

WordMapper Class

package com.company;

import org.apache.hadoop.io.IntWritable;

import org.apache.hadoop.io.LongWritable;

import org.apache.hadoop.io.Text;

import org.apache.hadoop.mapreduce.Mapper;

import java.io.IOException;

Hadoop has it’s own set of

basic types optimized for

net work serialization

public class WordCountMapper extends Mapper<LongWritable,Text,Text,IntWritable>

{

@Override

public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException{

String line = value.toString();

for (String word : line.split(" ")){

if(word.length()>0){

context.write(new Text(word),new IntWritable(1));

}

WordMapper Class

package com.company;

These are wrappers around

Java primitive types

import org.apache.hadoop.io.IntWritable;

import org.apache.hadoop.io.LongWritable;

import org.apache.hadoop.io.Text;

import org.apache.hadoop.mapreduce.Mapper;

import java.io.IOException;

public class WordCountMapper extends Mapper<LongWritable,Text,Text,IntWritable> {

@Override

public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException{

String line = value.toString();

for (String word : line.split(" ")){

if(word.length()>0){

context.write(new Text(word),new IntWritable(1));

}

WordMapper Class

package com.company;

These types are implemented

using the Writable Interface

import org.apache.hadoop.io.IntWritable;

import org.apache.hadoop.io.LongWritable;

import org.apache.hadoop.io.Text;

import org.apache.hadoop.mapreduce.Mapper;

import java.io.IOException;

public class WordCountMapper extends Mapper<LongWritable,Text,Text,IntWritable> {

@Override

public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException{

String line = value.toString();

for (String word : line.split(" ")){

if(word.length()>0){

context.write(new Text(word),new IntWritable(1));

}

WordMapper Class

package com.company;

Corresponding Hadoop type

for Java Long

import org.apache.hadoop.io.IntWritable;

import org.apache.hadoop.io.LongWritable;

import org.apache.hadoop.io.Text;

import org.apache.hadoop.mapreduce.Mapper;

import java.io.IOException;

public class WordCountMapper extends Mapper<LongWritable,Text,Text,IntWritable> {

@Override

public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException{

String line = value.toString();

for (String word : line.split(" ")){

if(word.length()>0){

context.write(new Text(word),new IntWritable(1));

}

WordMapper Class

package com.company;

Corresponding Hadoop type

for Java String

import org.apache.hadoop.io.IntWritable;

import org.apache.hadoop.io.LongWritable;

import org.apache.hadoop.io.Text;

import org.apache.hadoop.mapreduce.Mapper;

import java.io.IOException;

public class WordCountMapper extends Mapper<LongWritable,Text,Text,IntWritable> {

@Override

public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException{

String line = value.toString();

for (String word : line.split(" ")){

if(word.length()>0){

context.write(new Text(word),new IntWritable(1));

}

package com.company;

Corresponding Hadoop type

for Java Integer

import org.apache.hadoop.io.IntWritable;

import org.apache.hadoop.io.LongWritable;

import org.apache.hadoop.io.Text;

import org.apache.hadoop.mapreduce.Mapper;

import java.io.IOException;

public class WordCountMapper extends Mapper<LongWritable,Text,Text,IntWritable> {

@Override

public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException{

String line = value.toString();

for (String word : line.split(" ")){

if(word.length()>0){

context.write(new Text(word),new IntWritable(1));

}

WordMapper Class

package com.company;

import org.apache.hadoop.io.IntWritable;

import org.apache.hadoop.io.LongWritable;

import org.apache.hadoop.io.Text;

import org.apache.hadoop.mapreduce.Mapper;

import java.io.IOException;

In our example, the input

to map has the form

public class WordCountMapper extends Mapper<LongWritable,Text,Text,IntWritable>

{

@Override

public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException{

String line = value.toString();

for (String word : line.split(" ")){

if(word.length()>0){

context.write(new Text(word),new IntWritable(1));

}

WordMapper Class

package com.company;

import org.apache.hadoop.io.IntWritable;

import org.apache.hadoop.io.LongWritable;

import org.apache.hadoop.io.Text;

import org.apache.hadoop.mapreduce.Mapper;

import java.io.IOException;

In our example, the output

of map has the form

public class WordCountMapper extends Mapper<LongWritable,Text,Text,IntWritable>

{

@Override

public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException{

String line = value.toString();

for (String word : line.split(" ")){

if(word.length()>0){

context.write(new Text(word),new IntWritable(1));

}

WordMapper Class

package com.company;

import org.apache.hadoop.io.IntWritable;

import org.apache.hadoop.io.LongWritable;

import org.apache.hadoop.io.Text;

import org.apache.hadoop.mapreduce.Mapper;

import java.io.IOException;

The Mapper class has an

abstract method called map()

public class WordCountMapper extends Mapper<LongWritable,Text,Text,IntWritable> {

@Override

public void map(LongWritable key, Text value, Context context) throws

IOException, InterruptedException {

String line = value.toString();

for (String word : line.split(" ")){

if(word.length()>0){

context.write(new Text(word),new IntWritable(1));

}

WordMapper Class

package com.company;

import org.apache.hadoop.io.IntWritable;

import org.apache.hadoop.io.LongWritable;

import org.apache.hadoop.io.Text;

import org.apache.hadoop.mapreduce.Mapper;

import java.io.IOException;

We’ll override this method

with our own implementation

public class WordCountMapper extends Mapper<LongWritable,Text,Text,IntWritable> {

@Override

public void map(LongWritable key, Text value, Context context) throws

IOException, InterruptedException {

String line = value.toString();

for (String word : line.split(" ")){

if(word.length()>0){

context.write(new Text(word),new IntWritable(1));

}

WordMapper Class

package com.company;

import org.apache.hadoop.io.IntWritable;

import org.apache.hadoop.io.LongWritable;

import org.apache.hadoop.io.Text;

import org.apache.hadoop.mapreduce.Mapper;

import java.io.IOException;

The map() method

takes a Key and a Value

public class WordCountMapper extends Mapper<LongWritable,Text,Text,IntWritable> {

@Override

public void map(LongWritable key, Text value, Context context) throws

IOException, InterruptedException {

String line = value.toString();

for (String word : line.split(" ")){

if(word.length()>0){

context.write(new Text(word),new IntWritable(1));

}

WordMapper Class

package com.company;

import org.apache.hadoop.io.IntWritable;

import org.apache.hadoop.io.LongWritable;

import org.apache.hadoop.io.Text;

import org.apache.hadoop.mapreduce.Mapper;

import java.io.IOException;

It processes the input and writes

the output to a Context object

public class WordCountMapper extends Mapper<LongWritable,Text,Text,IntWritable> {

@Override

public void map(LongWritable key, Text value, Context context) throws

IOException, InterruptedException {

String line = value.toString();

for (String word : line.split(" ")){

if(word.length()>0){

context.write(new Text(word),new IntWritable(1));

}

WordMapper Class

package com.company;

import org.apache.hadoop.io.IntWritable;

import org.apache.hadoop.io.LongWritable;

import org.apache.hadoop.io.Text;

import org.apache.hadoop.mapreduce.Mapper;

import java.io.IOException;

The Context object stores the

output and is accessed by the rest

of the MapReduce system

public class WordCountMapper extends Mapper<LongWritable,Text,Text,IntWritable> {

@Override

public void map(LongWritable key, Text value, Context context) throws

IOException, InterruptedException {

...

MapReduce Overview and Implementation

MapReduce Overview and Implementation