How to use Spark's repartitionAndSortWithinPartitions?
<p>I am trying to build a minimal working example of <code>repartitionAndSortWithinPartitions</code> in order to understand the function. I have got so far (not working, the distinct throws the values around so that they get out of order)</p>

<pre><code>def partval(partID:Int, iter: Iterator[Int]): Iterator[Tuple2[Int, Int]] = {
  iter.map( x =&gt; (partID, x)).toList.iterator
}

val part20to3_chaos = sc.parallelize(1 to 20, 3).distinct
val part20to2_sorted = part20to3_chaos.repartitionAndSortWithinPartitions(2)
part20to2_sorted.mapPartitionsWithIndex(partval).collect
</code></pre>

<p>but get the error</p>

<pre><code>Name: Compile Error
Message: &lt;console&gt;:22: error: value repartitionAndSortWithinPartitions is not a member of org.apache.spark.rdd.RDD[Int]
             val part20to2_sorted = part20to3_chaos.repartitionAndSortWithinPartitions(2)
</code></pre>

<p>I tried using the <a href="http://spark.apache.org/docs/latest/api/scala/index.html" rel="noreferrer">scaladoc</a>, but wasn't able to find which class provides <code>repartitionAndSortWithinPartitions</code>. (Btw: This scaladoc is not impressive: Why is <code>MapPartitionsRDD</code> missing? How can I search for a method?)</p>

<p>Realising I need a partitioner object, next I tried to</p>

<pre><code>val rangePartitioner = new org.apache.spark.RangePartitioner(2, part20to3_chaos)
val part20to2_sorted = part20to3_chaos.repartitionAndSortWithinPartitions(rangePartitioner)
part20to2_sorted.mapPartitionsWithIndex(partval).collect
</code></pre>

<p>but got</p>

<pre><code>Name: Compile Error
Message: &lt;console&gt;:22: error: type mismatch;
 found   : org.apache.spark.rdd.RDD[Int]
 required: org.apache.spark.rdd.RDD[_ &lt;: Product2[?,?]]
Error occurred in an application involving default arguments.
         val rPartitioner = new org.apache.spark.RangePartitioner(2, part20to3_chaos)
</code></pre>

<p>How do I get this to compile? Could I get a working example, please?</p>