How to un-cube, un-rollup, or flatten hierarchical/nested DataFrame from JSON?
<p>I have a spark dataframe built from JSON that has a schema like this:</p>

<pre><code>root
|-- Engagement ID: string (nullable = true)
|-- Transcript: array (nullable = true)
|    |-- element: struct (containsNull = true)
|    |    |-- created_at: string (nullable = true)
|    |    |-- message: string (nullable = true)
|    |    |-- sender: struct (nullable = true)
|    |    |    |-- href: string (nullable = true)
|    |    |    |-- name: string (nullable = true)
|    |    |    |-- type: string (nullable = true)
|-- _corrupt_record: string (nullable = true)
</code></pre>

<p>What I'd like to do though is flatten it out, at least to have the messages in each transcript each on their own row, essentially to something like:</p>

<pre><code>root
|-- Engagement ID: string (nullable = true)
|-- created_at: string (nullable = true)
|-- message: string (nullable = true)
|-- sender: struct (nullable = true)
|-- name: string (nullable = true)
|-- type: string (nullable = true)
</code></pre>

<p>Of course this will end up with lots of rows with the same Engagement ID, and the original structure is something like what you would get doing a rollup or cube. Is there a straightforward way to uncube, unrollup, flatten, or whatever the term might be? Is it possible without using the underlying RDDs?</p>