Rolling your own reduceByKey in Spark Dataset
<p>I'm trying to learn to use DataFrames and DataSets more in addition to RDDs.  For an RDD, I know I can do <code>someRDD.reduceByKey((x,y) =&gt; x + y)</code>, but I don't see that function for Dataset.  So I decided to write one. </p>

<pre><code>someRdd.map(x =&gt; ((x.fromId,x.toId),1)).map(x =&gt; collection.mutable.Map(x)).reduce((x,y) =&gt; {
  val result = mutable.HashMap.empty[(Long,Long),Int]
  val keys = mutable.HashSet.empty[(Long,Long)]
  y.keys.foreach(z =&gt; keys += z)
  x.keys.foreach(z =&gt; keys += z)
  for (elem &lt;- keys) {
    val s1 = if(x.contains(elem)) x(elem) else 0
    val s2 = if(y.contains(elem)) y(elem) else 0
    result(elem) = s1 + s2
  }
  result
})
</code></pre>

<p>However, this returns everything to the driver.  How would you write this to return a <code>Dataset</code>?  Maybe mapPartition and do it there?</p>

<p>Note this compiles but does not run because it doesn't have encoders for <code>Map</code> yet</p>