Why does Spark save Map phase output to local disk?
<p>I'm trying to understand spark shuffle process deeply. When i start reading i came across the following point.</p>

<blockquote>
  <p>Spark writes the Map task(ShuffleMapTask) output directly to disk on completion.</p>
</blockquote>

<p>I would like to understand the following w.r.t to Hadoop MapReduce.</p>

<ol>
<li><p>If both Map-Reduce and Spark writes the data to the local disk then how spark shuffle process is different from Hadoop MapReduce?</p></li>
<li><p>Since data is represented as RDD's in Spark why don't these outputs remain in the node executors memory?</p></li>
<li><p>How is the output of the Map tasks from Hadoop MapReduce and Spark different?</p></li>
<li><p>If there are lot of small intermediate files as output how spark handles the network and I/O bottleneck?</p></li>
</ol>