Spark dataframes convert nested JSON to seperate columns
<p>I've a stream of JSONs with following structure that gets converted to dataframe     </p>

<pre><code>{
  "a": 3936,
  "b": 123,
  "c": "34",
  "attributes": {
    "d": "146",
    "e": "12",
    "f": "23"
  }
}
</code></pre>

<p>The dataframe show functions results in following output</p>

<pre><code>sqlContext.read.json(jsonRDD).show

+----+-----------+---+---+
|   a| attributes|  b|  c|
+----+-----------+---+---+
|3936|[146,12,23]|123| 34|
+----+-----------+---+---+
</code></pre>

<p>How can I split attributes column (nested JSON structure) into <strong>attributes.d, attributes.e and attributes.f</strong> as <em>seperate</em> columns into a new dataframe, so I can have columns as a, b, c, attributes.d, attributes.e and attributes.f in the new dataframe? </p>