Hadoop Streaming の使い方 mapper を perl で書くサンプル

このドキュメントの内容は、以下の通りです。

• perl1.pl
• perl1.sh
• input
• 実行
• コマンド

perl1.pl

#!/usr/local/bin/perl -w

use strict;

sub main
{
	while (<STDIN>)
	{
		chomp;

		# ここに処理

		print $_ . "\t1\n";
	}
	exit (0);
}

& main;

perl1.sh

#!/bin/sh

HADOOP_HOME=/home/kaworu/Hadoop/hadoop-1.0.3

HADOOP=$HADOOP_HOME/bin/hadoop

STREAMING=$HADOOP_HOME/contrib/streaming/hadoop-streaming-1.0.3.jar

$HADOOP jar $STREAMING \
	-input input -output output \
	-mapper $PWD/perl1.pl -reducer /usr/bin/wc

input

$ ls input/
a
$ cat input/a
1
2
3

実行

$ sh perl1.sh
packageJobJar: [/tmp/hadoop-kaworu/hadoop-unjar45233/] [] /var/tmp/streamjob45234.jar tmpDir=null
12/07/03 01:08:06 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
12/07/03 01:08:06 WARN snappy.LoadSnappy: Snappy native library not loaded
12/07/03 01:08:06 INFO mapred.FileInputFormat: Total input paths to process : 1
12/07/03 01:08:07 INFO streaming.StreamJob: getLocalDirs(): [/tmp/hadoop-kaworu/mapred/local]
12/07/03 01:08:07 INFO streaming.StreamJob: Running job: job_201207281237_0042
12/07/03 01:08:07 INFO streaming.StreamJob: To kill this job, run:
12/07/03 01:08:07 INFO streaming.StreamJob: /home/kaworu/Hadoop/hadoop-1.0.3/libexec/../bin/hadoop job  -Dmapred.job.tracker=localhost:9001 -kill job_201207281237_0042
12/07/03 01:08:07 INFO streaming.StreamJob: Tracking URL: http://localhost:50030/jobdetails.jsp?jobid=job_201207281237_0042
12/07/03 01:08:08 INFO streaming.StreamJob:  map 0%  reduce 0%
12/07/03 01:08:26 INFO streaming.StreamJob:  map 100%  reduce 0%
12/07/03 01:08:38 INFO streaming.StreamJob:  map 100%  reduce 100%
12/07/03 01:08:44 INFO streaming.StreamJob: Job complete: job_201207281237_0042
12/07/03 01:08:44 INFO streaming.StreamJob: Output: output

コマンド

./bin/hadoop dfs -ls output/part-00000
./bin/hadoop dfs -cat output/part-00000
./bin/hadoop dfs -rmr output