How to optimize groupBy() operation on Spark RDD
<p>My question is about alternatives/optimization to <code>groupBy()</code> operation on RDD. I have millions of <code>Message</code> instances which needs to be grouped based on some ID. I use <code>groupBy()</code> to accomplish the goal but I know the fact that it is  an expensive one and it takes huge amount of processing time .</p>

<p>So I tried <code>reduceByKey(func)</code> and <code>combineByKey()</code> as an alternative which was suggested in some of resources that I went through but its not a fit to my scenario where 
 <strong><em>My requirement is just grouping</em></strong>. But my data size is huge where <code>groupBy</code> consumes more <strong>than 70% of my job's time</strong>.So any suggestions or solution are appreciated.  </p>

<p><strong>My input Data</strong></p>

<blockquote>
  <p>Seq(new Message("1", "abc", "Request"), new Message("1", "cba",
  "Response"), new Message("2", "def", "Request"), new Message("2",
  "fed", "Response"), new Message("3", "ghi", "Request"), new
  Message("3", "ihg", "Response"))</p>
</blockquote>

<p>For the sake of simplicity, i have given input data in <code>Seq</code> </p>

<pre><code> def groupMessages(sourceRdd: RDD[Message]): Unit {

    val messageIdRdd = sourceRdd.mapPartitions(partitionData =&gt; {
      partitionData.map(row =&gt; (row.uniqueID(), row))

    })

    /*
     Group messages based on its message id. This functionality is to combine request response
     together as single transaction based on message id.
     */
    val groupbyRdd = messageIdRdd.groupBy(_._1) 
}
</code></pre>

<p><strong>Expected Result</strong> </p>

<blockquote>
  <p>(2,CompactBuffer((2,Message(2,def,Request)),
  (2,Message(2,fed,Response))))
  (3,CompactBuffer((3,Message(3,ghi,Request)),
  (3,Message(3,ihg,Response))))
  (1,CompactBuffer((1,Message(1,abc,Request)),
  (1,Message(1,cba,Response))))</p>
</blockquote>