spark dataframe keep most recent record
<p>I have a dataframe similar to: </p>

<pre><code>id  | date       | value
--- | ---------- | ------
1   | 2016-01-07 | 13.90
1   | 2016-01-16 | 14.50
2   | 2016-01-09 | 10.50
2   | 2016-01-28 | 5.50
3   | 2016-01-05 | 1.50
</code></pre>

<p>I am trying to keep the most recent values for each id, like this:</p>

<pre><code>id  | date       | value
--- | ---------- | ------
1   | 2016-01-16 | 14.50
2   | 2016-01-28 | 5.50
3   | 2016-01-05 | 1.50
</code></pre>

<p>I have tried sort by date desc and after drop duplicates:</p>

<pre><code>new_df = df.orderBy(df.date.desc()).dropDuplicates(['id'])   
</code></pre>

<p>My questions are, <code>dropDuplicates()</code> will keep the first duplicate value that it finds? and is there a better way to accomplish what I want to do? By the way, I'm using python.</p>

<p>Thank you.</p>