Apply a custom function to a spark dataframe group
<p>I have a very big table of time series data that have these columns:</p>

<ul>
<li>Timestamp</li>
<li>LicensePlate</li>
<li>UberRide#</li>
<li>Speed</li>
</ul>

<p>Each collection of LicensePlate/UberRide data should be processed considering the whole set of data. In others words, I <strong>do not</strong> need to proccess the data row by row, but all rows grouped by (LicensePlate/UberRide) together.</p>

<p>I am planning to use spark with dataframe api, but I am confused on how can I perform a custom calculation over spark grouped dataframe.</p>

<p>What I need to do is:</p>

<ol>
<li>Get all data</li>
<li>Group by some columns</li>
<li>Foreach spark dataframe group apply a f(x). Return a custom object foreach group</li>
<li>Get the results by applying g(x) and returning a single custom object</li>
</ol>

<p>How can I do steps 3 and 4? Any hints over which spark API (dataframe, dataset, rdd, maybe pandas...) should I use?</p>

<p>The whole workflow can be seen below:</p>

<p><a href="https://i.stack.imgur.com/9XsB4.png" rel="noreferrer"><img src="https://i.stack.imgur.com/9XsB4.png" alt="Workflow"></a></p>