Spark: difference of semantics between reduce and reduceByKey
<p>In Spark's documentation, it says that RDDs method <a href="http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.rdd.RDD" rel="nofollow"><code>reduce</code></a> requires a associative AND commutative binary function.</p>

<p>However, the method <a href="http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.rdd.PairRDDFunctions" rel="nofollow"><code>reduceByKey</code></a> ONLY requires an associative binary function.</p>

<pre><code>sc.textFile("file4kB", 4)
</code></pre>

<p>I did some tests, and apparently it's the behavior I get. Why this difference? Why does <code>reduceByKey</code> ensure the binary function is always applied in certain order (to accommodate for the lack of commutativity) when <code>reduce</code> does not?</p>

<p>Example, if a load some (small) text with 4 partitions (minimum):</p>

<pre><code>val r = sc.textFile("file4k", 4)
</code></pre>

<p>then:</p>

<pre><code>r.reduce(_ + _)
</code></pre>

<p>returns a string where parts are not always in the same order, whereas:</p>

<pre><code>r.map(x =&gt; (1,x)).reduceByKey(_ + _).first
</code></pre>

<p>always returns the same string (where everything is in the same order than in the original file).</p>

<p>(I checked with <code>r.glom</code> and the file content is indeed spread over 4 partitions, there is no empty partition).</p>