How to sort JavaRDD<Row> by multiple fields and just reserve the particular data in Java Spark
<p>I have an input data with type <code>JavaRDD&lt;Row&gt;</code>.
The <code>Row</code> has two fields.</p>

<pre><code>[
  {"fieldName":"requestId", "fieldType":"String"}, 
  {"fieldName":"price", "fieldType":"double"}
]
</code></pre>

<p>The <code>requestId</code> and <code>price</code> could be duplicated in many <code>Rows</code>. My purpose is to just reserve the <code>Row</code> with max <code>price</code> from those <code>Rows</code> with the same <code>requestId</code>. Actually, any methods will be ok even not use sort.</p>

<p>For example, the input is like this:</p>

<pre><code>76044601-8029-4e09-9708-41dd125ae4bb    1676.304091136485
76044601-8029-4e09-9708-41dd125ae4bb    3898.9987591932413
ad0acb4a-100d-4624-b863-fcf275ce28db    7518.603722172683
76044601-8029-4e09-9708-41dd125ae4bb    3308.4421575701463
26f639bc-2041-435c-86da-73b997c0cc64    1737.7186292370193
beeb7fc1-2a2d-4943-8237-c281ee7c9617    4941.882928279789
26f639bc-2041-435c-86da-73b997c0cc64    1710.328581775302
</code></pre>

<p>The output data should be like this(the output order is not a problem):</p>

<pre><code>76044601-8029-4e09-9708-41dd125ae4bb    3898.9987591932413
ad0acb4a-100d-4624-b863-fcf275ce28db    7518.603722172683
26f639bc-2041-435c-86da-73b997c0cc64    1737.7186292370193
beeb7fc1-2a2d-4943-8237-c281ee7c9617    4941.882928279789
</code></pre>

<p>Candidate method:</p>

<pre><code>JavaRDD&lt;Row&gt; javaRDD = dataFrame.toJavaRDD().mapToPair(new PairFunction&lt;Row, String, Row&gt;() {
        @Override
        public Tuple2&lt;String, Row&gt; call(Row row) {
            String key = String.valueOf(row.getAs("requestid"));
            return new Tuple2&lt;String, Row&gt;(key, row);
        }
    }).reduceByKey(new Function2&lt;Row, Row, Row&gt;() {
        @Override
        public Row call(Row row1, Row row2) throws Exception {
            double rs1 = Double.parseDouble(String.valueOf(row1.getAs("price")));
            double rs2 = Double.parseDouble(String.valueOf(row2.getAs("price")));
            if (rs1 &lt; rs2) {
                return row2;
            } else {
                return row1;
            }
        }
    }).map(new Function&lt;Tuple2&lt;String, Row&gt;, Row&gt;() {
        @Override
        public Row call(Tuple2&lt;String, Row&gt; tuple) {
            return tuple._2;
        }
    });
</code></pre>