Spark dataframe grouping, sorting, and selecting top rows for a set of columns
<p>I am using Spark 1.5.0. I have a Spark dataframe with following columns:</p>

<pre><code>| user_id | description | fName | weight |
</code></pre>

<p>What I would like to do is to select top 10 and bottom 10 rows (based on the value of column weight, which is of datatype Double) per user. How do I do that using Spark SQL or dataframe operations?</p>

<p>For example. for simplicity I am only selecting top 2 rows (based on weight) per user. I would like to sort the o/p on the value of the absolute weight.</p>

<pre><code>u1  desc1 f1  -0.20
u1  desc1 f1  +0.20
u2  desc1 f1  0.80
u2  desc1 f1  -0.60
u1  desc1 f1  1.10
u1  desc1 f1  6.40
u2  desc1 f1  0.05
u1  desc1 f1  -3.20
u2  desc1 f1  0.50
u2  desc1 f1  -0.70
u2  desc1 f1  -0.80   
</code></pre>

<p>Here is the desired o/p:</p>

<pre><code>u1  desc1 f1  6.40
u1  desc1 f1  -3.20
u1  desc1 f1  1.10
u1  desc1 f1  -0.20
u2  desc1 f1  0.80
u2  desc1 f1  -0.80
u2  desc1 f1  -0.70
u2  desc1 f1  0.50
</code></pre>