Spark Scala: Aggregate DataFrame Column Values into a Ordered List
<p>I have a spark scala DataFrame that has four values: (id, day, val, order). I want to create a new DataFrame with columns: (id, day, value_list: List(val1, val2, ..., valn)) where val1, through valn are ordered by asc order value. </p>

<p>For instance: </p>

<pre><code>(50, 113, 1, 1), 
(50, 113, 1, 3), 
(50, 113, 2, 2), 
(51, 114, 1, 2), 
(51, 114, 2, 1), 
(51, 113, 1, 1)
</code></pre>

<p>would become:</p>

<pre><code>((51,113),List(1))
((51,114),List(2, 1)
((50,113),List(1, 2, 1))
</code></pre>

<p>I'm close, but don't know what to do after I've aggregated the data into a list. I'm not sure how to then have spark order each value list by the order int:</p>

<pre><code>import org.apache.spark.sql.Row

val testList = List((50, 113, 1, 1), (50, 113, 1, 3), (50, 113, 2, 2), (51, 114, 1, 2), (51, 114, 2, 1), (51, 113, 1, 1))
val testDF = sqlContext.sparkContext.parallelize(testList).toDF("id1", "id2", "val", "order")

val rDD1 = testDF.map{case Row(key1: Int, key2: Int, val1: Int, val2: Int)  =&gt; ((key1, key2), List((val1, val2)))}
val rDD2 = rDD1.reduceByKey{case (x, y) =&gt;  x ++ y}
</code></pre>

<p>where the output looks like: </p>

<pre><code>((51,113),List((1,1)))
((51,114),List((1,2), (2,1)))
((50,113),List((1,3), (1,1), (2,2)))
</code></pre>

<p>The next step would be to produce:</p>

<pre><code>((51,113),List((1,1)))
((51,114),List((2,1), (1,2)))
((50,113),List((1,1), (2,2), (1,3)))
</code></pre>