Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:  Switch to Threaded View
MapReduce, mail # dev - Why In-memory Mapoutput is necessary in
ReduceCopier  Copy link to this message Why In-memory Mapoutput
is necessary in ReduceCopier Ling Kun 2013-03-11, 10:27 Dear
all,       I am focusing on the Mapoutput copier implementation.
This part of code will try to get mapoutputs, and merge them
into a file that can feed to reduce functions. I have the
following questions.  1. All the local file mapoutput data will
be merged together by the LocalFSMerge, and the in-memory mapout
will be merged by InMemFSMergeThread. For the
InMemFSMergeThread, there is also a writer object   which write
the result to outputPath ( ReduceTask.java Line 2843). It seems
after merging, in-memory mapoutput and local file mapoutput data
will all be stored in local file system. Why not just using the
local file for all mapoutput data.  2. After using http to get
 some fragment of a map output