Spark remove duplicate rows from DataFrame
<p>Assume that I am having a DataFrame like : </p>

<pre><code>val json = sc.parallelize(Seq("""{"a":1, "b":2, "c":22, "d":34}""","""{"a":3, "b":9, "c":22, "d":12}""","""{"a":1, "b":4, "c":23, "d":12}"""))
val df = sqlContext.read.json(json)
</code></pre>

<p>I want to remove duplicate rows for column "a" based on the value of column "b". i.e, if there are duplicate rows for column "a", I want to keep the one with larger value for "b". For the above example, after processing, I need only </p>

<blockquote>
  <p>{"a":3, "b":9, "c":22, "d":12}</p>
</blockquote>

<p>and </p>

<blockquote>
  <p>{"a":1, "b":4, "c":23, "d":12}</p>
</blockquote>

<p>Spark DataFrame dropDuplicates API doesn't seem to support this. With the RDD approach, I can do a <code>map().reduceByKey()</code>, but what DataFrame specific operation is there to do this?</p>

<p>Appreciate some help, thanks.</p>