Spark - union dataframe rows into one row
<p>I have dataframe that contains rows with identical id.
I need to union all the rows with the same id into one row (one json)</p>

<p>Here is example of the data:</p>

<pre><code>id  first_name   last_name
1    JAMES         SMITH
2    MARY          BROWN
2    DAVID         WILLIAMS
1    ROBERT        DAVIS
</code></pre>

<p>the requested result is:</p>

<pre><code>{
  id:1,
  entities: [{
    first_name:JAMES,
    last_name:SMITH 
   }, {
    first_name:ROBERT,
    last_name:DAVIS
  }]
}
{
  id:2,
  entities: [{
    first_name:MARY,
    last_name:BROWN 
   }, {
    first_name:DAVID,
    last_name:WILLIAMS
  }]
}  
</code></pre>

<p>Can it be done?</p>

<p>Regards, Yaniv</p>