SparkSQL datetime function
<p>I am using Spark 1.3. I want to do some date-based calculations. In the following dataset, for each unique id, I want to get the record where beging_date is maximum (the latest record).</p>
<p>Also, when I read the data from file, should I cast it as TimestampType (import org.apache.spark.sql.types.TimestampType)?</p>
<p>Here is some sample data:</p>
<blockquote>
<p>id    beging_date end_date</p>
<p>1    1/1/2016  20:06:00.0   1/4/2016    20:06:00.0</p>
<p>2    1/5/2013   20:06:00.0  1/8/2016    20:06:00.0</p>
<p>1    1/6/2013  20:06:00.0 1/18/2016   20:06:00.0</p>
<p>3    2/1/2013   20:06:00.0  2/5/2016   20:06:00.0</p>
<p>1    1/20/2013  20:06:00.0  2/4/2016 20:06:00.0</p>
<p>3    3/5/2013  20:06:00.0    3/8/2016   20:06:00.0</p>
</blockquote>
<p>Here is the desired output:</p>
<blockquote>
<p>id    beging_date end_date</p>
<p>1    1/20/2013 20:06:00.0  2/4/2016   20:06:00.0</p>
<p>2    1/5/2013  20:06:00.0   1/8/2016 20:06:00.0</p>
<p>3    3/5/2013  20:06:00.0    3/8/2016   20:06:00.0</p>
</blockquote>