{"id":491,"date":"2021-02-19T16:35:33","date_gmt":"2021-02-19T07:35:33","guid":{"rendered":"https:\/\/www.opentone.co.jp\/optonlabos\/?p=491"},"modified":"2025-05-12T10:06:22","modified_gmt":"2025-05-12T01:06:22","slug":"hadoop%e5%ad%a6%e7%bf%92%ef%bd%9e%e5%8b%95%e4%bd%9c%ef%bd%9e","status":"publish","type":"post","link":"https:\/\/www.opentone.co.jp\/ot-lab\/all\/uncategorized\/hadoop%e5%ad%a6%e7%bf%92%ef%bd%9e%e5%8b%95%e4%bd%9c%ef%bd%9e","title":{"rendered":"Hadoop\u5b66\u7fd2\uff5e\u52d5\u4f5c\uff5e"},"content":{"rendered":"\n

\u3069\u3046\u3082\u3002koglen\u3067\u3059\u3002
\u524d\u56de\u306e\u300cHadoop\u300d\u57fa\u790e\u5b66\u7fd2\u306b\u3064\u3065\u3044\u3066\u74b0\u5883\u52d5\u4f5c\u306e\u89b3\u70b9\u3067\u5b66\u7fd2\u3057\u305f\u3053\u3068\u3092\u66f8\u3044\u3066\u307f\u305f\u3044\u3068\u601d\u3044\u307e\u3059\u3002<\/p>\n\n\n\n

\u672c\u6765\u3067\u3042\u308c\u3070\u4e26\u5217\u51e6\u7406\u3067\u300cHadoop\u300d\u306e\u8ca0\u8377\u5206\u6563\u51e6\u7406\u306e\u5999\u6280\u3092\u5473\u308f\u3044\u305f\u3044\u3068\u3053\u308d\u3067\u3059\u304c\u3001
\u4eca\u56de\u306f\u30ed\u30fc\u30ab\u30ebPC\u306b\u300cHadoop\u300d\u3092\u30a4\u30f3\u30b9\u30c8\u30fc\u30eb\u3057MapReduce\u306e\u30b5\u30f3\u30d7\u30eb\u30d7\u30ed\u30b0\u30e9\u30e0\u3092\u5b9f\u65bd\u3057\u3066\u307f\u305f\u3044\u3068\u601d\u3044\u307e\u3059\u3002
\u300cHadoop\u300d\u306fUnix\u74b0\u5883\u3067\u52d5\u4f5c\u3059\u308b\u305f\u3081\u3001Windows\u4e0a\u3067\u52d5\u4f5c\u3055\u305b\u308b\u306b\u306fCygwin<\/a>\u306e\u30a4\u30f3\u30b9\u30c8\u30fc\u30eb\u304c\u5fc5\u8981\u306b\u306a\u308a\u307e\u3059\u3002
\u30ed\u30fc\u30ab\u30ebPC\u306b\u306f\u65e2\u306bJava\u306f\u30a4\u30f3\u30b9\u30c8\u30fc\u30eb\u3055\u308c\u3066\u3044\u308b\u305f\u3081\u3001Cygwin\u3068\u300cHadoop<\/a>\u300d\u306e\u30a4\u30f3\u30b9\u30c8\u30fc\u30eb\u3092\u884c\u3044\u307e\u3059\u3002<\/p>\n\n\n\n

Cygwin\u3092\u30a4\u30f3\u30b9\u30c8\u30fc\u30eb\u3057\u305f\u5f8c\u3001\u300cHadoop\u300d\u3092Cygwin\u3092\u30a4\u30f3\u30b9\u30c8\u30fc\u30eb\u3057\u305f\u5148\u306eusr\/local\/\u914d\u4e0b\u306b\u30a4\u30f3\u30b9\u30c8\u30fc\u30eb\u3057\u3066\u52d5\u4f5c\u691c\u8a3c\u3092\u884c\u3044\u307e\u3057\u305f\u3002
\u691c\u8a3c\u3059\u308b\u30d7\u30ed\u30b0\u30e9\u30e0\u306fMapReduce\u306e\u30ef\u30fc\u30c9\u30ab\u30a6\u30f3\u30c8\u51e6\u7406\u3067\u3001\u5bfe\u8c61\u3068\u3057\u3066\u6307\u5b9a\u3055\u308c\u305f\u30d5\u30a1\u30a4\u30eb\u3084\u30c7\u30a3\u30ec\u30af\u30c8\u30ea\u914d\u4e0b\u306e\u51fa\u73fe\u3059\u308b\u3059\u3079\u3066\u306e\u30ef\u30fc\u30c9\u3068\u305d\u306e\u51fa\u73fe\u56de\u6570\u3092\u30ab\u30a6\u30f3\u30c8\u3059\u308b\u30d7\u30ed\u30b0\u30e9\u30e0\u3067\u3059\u3002
hoge.txt\u306b\u4ee5\u4e0b\u306e\u3088\u3046\u306a\u6587\u5b57\u5217\u3092\u8a18\u8f09\u3057output\u30d5\u30a9\u30eb\u30c0\u306b\u30ab\u30a6\u30f3\u30c8\u7d50\u679c\u3092\u51fa\u529b\u3057\u3066\u307f\u307e\u3059\u3002<\/p>\n\n\n\n

\u25a0hoge.txt
hoge
abc def
ghi
jkl
mno pqe stu vwx
yz
hoge hoge<\/p><\/blockquote>\n\n\n\n

\u25a0\u5b9f\u884c\u30b3\u30de\u30f3\u30c9
.\/bin\/hadoop jar hadoop-mapred-examples-0.21.0.jar wordcount hoge.txt .\/output<\/p><\/blockquote>\n\n\n\n

\u25a0\u30b3\u30f3\u30bd\u30fc\u30eb\u30ed\u30b0
XX\/XX\/XX XX:XX:XX INFO mapreduce.Job: Counters: 18
FileInputFormatCounters
BYTES_READ=55
FileSystemCounters
FILE_BYTES_READ=504678
FILE_BYTES_WRITTEN=634892
Map-Reduce Framework
Combine input records=12
Combine output records=10
Failed Shuffles=0
GC time elapsed (ms)=17
Map input records=7
Map output bytes=98
Map output records=12
Merged Map outputs=0
Reduce input groups=10
Reduce input records=10
Reduce output records=10
Reduce shuffle bytes=0
Shuffled Maps =0
Spilled Records=20
SPLIT_RAW_BYTES=113<\/p>

\u25a0\u7d50\u679c\u30d5\u30a1\u30a4\u30eb
abc    1
def    1
ghi    1
hoge    3
jkl    1
mno    1
pqe    1
stu    1
vwx    1
yz    1<\/p><\/blockquote>\n\n\n\n

\u4e0a\u8a18\u306e\u3088\u3046\u306b\u60f3\u5b9a\u3069\u304a\u308a\u306e\u7d50\u679c\u304c\u8fd4\u3063\u3066\u304d\u307e\u3057\u305f\u3002
\u7d9a\u3044\u3066\u306fhoge.txt\u306e\u6700\u7d42\u884c\u306b1\u884c\u8ffd\u52a0\u3057grep\u51e6\u7406\u3092\u5b9f\u65bd\u3057\u3066\u307f\u307e\u3059\u3002<\/p>\n\n\n\n

\u25a0hoge.txt
hoge
abc def
ghi
jkl
mno pqe stu vwx
yz
hoge hoge
hogu hoga<\/p><\/blockquote>\n\n\n\n

\u25a0\u5b9f\u884c\u30b3\u30de\u30f3\u30c9
bin\/hadoop jar hadoop-mapred-examples-0.21.0.jar grep hoge.txt output2 ‘hog[a-z]’<\/p><\/blockquote>\n\n\n\n

\u25a0\u30b3\u30f3\u30bd\u30fc\u30eb\u30ed\u30b0
XX\/XX\/XX XX:XX:XX INFO mapreduce.Job: Counters: 18
FileInputFormatCounters
BYTES_READ=149
FileSystemCounters
FILE_BYTES_READ=1327
FILE_BYTES_WRITTEN=253193
Map-Reduce Framework
Combine input records=0
Combine output records=0
Failed Shuffles=0
GC time elapsed (ms)=16
Map input records=3
Map output bytes=39
Map output records=3
Merged Map outputs=0
Reduce input groups=2
Reduce input records=3
Reduce output records=3
Reduce shuffle bytes=0
Shuffled Maps =0
Spilled Records=6
SPLIT_RAW_BYTES=138<\/p><\/blockquote>\n\n\n\n

\u25a0\u7d50\u679c\u30d5\u30a1\u30a4\u30eb
3    hoge
1    hogu
1    hoga<\/p><\/blockquote>\n\n\n\n

\u3053\u3061\u3089\u3082\u60f3\u5b9a\u901a\u308a\u3002<\/p>\n\n\n\n

\u300cHadoop\u300d\u306fOS\u3068\u5404\u7a2e\u74b0\u5883\u6574\u5099\u306e\u95a2\u4fc2\u304b\u3089\u6577\u5c45\u304c\u9ad8\u3044\u30a2\u30fc\u30ad\u30c6\u30af\u30c1\u30e3\u30fc\u3068\u611f\u3058\u3066\u3044\u307e\u3057\u305f\u304c\u3001\u5b9f\u969bWindows\u4e0a\u3067\u7c21\u5358\u306b\u30b5\u30f3\u30d7\u30eb\u3092\u52d5\u304b\u3059\u3053\u3068\u304c\u3067\u304d\u307e\u3057\u305f\u3002
\u3082\u3063\u3068\u672c\u683c\u7684\u306b\u52c9\u5f37\u3057\u305f\u3044\u305e\uff01\u3068\u3044\u3046\u65b9\u306f\u3001CDH<\/a>\uff08Cloudera’s Distribution for Hadoop\uff09\u3084Amazon EMR<\/a>\uff08Amazon Elastic MapReduce\uff09\u306a\u3069\u3092\u5229\u7528\u3055\u308c\u3066\u307f\u3066\u306f\u3044\u304b\u304c\u3067\u3057\u3087\u3046\u304b\uff1f <\/p>\n

\"\"<\/a><\/div>","protected":false},"excerpt":{"rendered":"

\u3069\u3046\u3082\u3002koglen\u3067\u3059\u3002\u524d\u56de\u306e\u300cHadoop\u300d\u57fa\u790e\u5b66\u7fd2\u306b\u3064\u3065\u3044\u3066\u74b0\u5883\u52d5\u4f5c\u306e\u89b3\u70b9\u3067\u5b66\u7fd2\u3057\u305f\u3053\u3068\u3092\u66f8\u3044<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"_uag_custom_page_level_css":"","_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"footnotes":""},"categories":[1],"tags":[],"class_list":["post-491","post","type-post","status-publish","format-standard","hentry","category-uncategorized",""],"acf":[],"aioseo_notices":[],"uagb_featured_image_src":{"full":false,"thumbnail":false,"medium":false,"medium_large":false,"large":false,"1536x1536":false,"2048x2048":false,"engage-mag-carousel-img":false,"engage-mag-carousel-img-landscape":false,"engage-mag-carousel-large-img":false,"engage-mag-carousel-large-img-landscape":false,"engage-mag-large-thumb":false,"engage-mag-small-thumb":false},"uagb_author_info":{"display_name":"ah106rx4o4","author_link":"https:\/\/www.opentone.co.jp\/ot-lab\/author\/ah106rx4o4"},"uagb_comment_info":0,"uagb_excerpt":"\u3069\u3046\u3082\u3002koglen\u3067\u3059\u3002\u524d\u56de\u306e\u300cHadoop\u300d\u57fa\u790e\u5b66\u7fd2\u306b\u3064\u3065\u3044\u3066\u74b0\u5883\u52d5\u4f5c\u306e\u89b3\u70b9\u3067\u5b66\u7fd2\u3057\u305f\u3053\u3068\u3092\u66f8\u3044","_links":{"self":[{"href":"https:\/\/www.opentone.co.jp\/ot-lab\/wp-json\/wp\/v2\/posts\/491","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.opentone.co.jp\/ot-lab\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.opentone.co.jp\/ot-lab\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.opentone.co.jp\/ot-lab\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.opentone.co.jp\/ot-lab\/wp-json\/wp\/v2\/comments?post=491"}],"version-history":[{"count":1,"href":"https:\/\/www.opentone.co.jp\/ot-lab\/wp-json\/wp\/v2\/posts\/491\/revisions"}],"predecessor-version":[{"id":492,"href":"https:\/\/www.opentone.co.jp\/ot-lab\/wp-json\/wp\/v2\/posts\/491\/revisions\/492"}],"wp:attachment":[{"href":"https:\/\/www.opentone.co.jp\/ot-lab\/wp-json\/wp\/v2\/media?parent=491"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.opentone.co.jp\/ot-lab\/wp-json\/wp\/v2\/categories?post=491"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.opentone.co.jp\/ot-lab\/wp-json\/wp\/v2\/tags?post=491"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}