Spark DataFrame: does groupBy after orderBy maintain that order?
<p>I have a Spark 2.0 dataframe <code>example</code> with the following structure:</p>

<pre><code>id, hour, count
id1, 0, 12
id1, 1, 55
..
id1, 23, 44
id2, 0, 12
id2, 1, 89
..
id2, 23, 34
etc.
</code></pre>

<p>It contains 24 entries for each id (one for each hour of the day) and is ordered by id, hour using the orderBy function. </p>

<p>I have created an Aggregator <code>groupConcat</code>:</p>

<pre><code>  def groupConcat(separator: String, columnToConcat: Int) = new Aggregator[Row, String, String] with Serializable {
    override def zero: String = ""

    override def reduce(b: String, a: Row) = b + separator + a.get(columnToConcat)

    override def merge(b1: String, b2: String) = b1 + b2

    override def finish(b: String) = b.substring(1)

    override def bufferEncoder: Encoder[String] = Encoders.STRING

    override def outputEncoder: Encoder[String] = Encoders.STRING
  }.toColumn
</code></pre>

<p>It helps me concatenate columns into strings to obtain this final dataframe:  </p>

<pre><code>id, hourly_count
id1, 12:55:..:44
id2, 12:89:..:34
etc.
</code></pre>

<p>My question is, if I do <code>example.orderBy($"id",$"hour").groupBy("id").agg(groupConcat(":",2) as "hourly_count")</code>, does that guarantee that the hourly counts will be ordered correctly in their respective buckets? </p>

<p>I read that this is not necessarily the case for RDDs (see <a href="https://stackoverflow.com/questions/29792320/spark-sort-by-key-and-then-group-by-to-get-ordered-iterable">Spark sort by key and then group by to get ordered iterable?</a>), but maybe it's different for DataFrames ? </p>

<p>If not, how can I work around it ? </p>