PySpark converting a column of type 'map' to multiple columns in a dataframe
<h2>Input</h2>
<p>I have a column <code>Parameters</code> of type <code>map</code> of the form:</p>
<pre><code>from pyspark.sql import SQLContext
sqlContext = SQLContext(sc)
d = [{'Parameters': {'foo': '1', 'bar': '2', 'baz': 'aaa'}}]
df = sqlContext.createDataFrame(d)

df.collect()
# [Row(Parameters={'foo': '1', 'bar': '2', 'baz': 'aaa'})]

df.printSchema()
# root
#  |-- Parameters: map (nullable = true)
#  |    |-- key: string
#  |    |-- value: string (valueContainsNull = true)
</code></pre>
<h2>Output</h2>
<p>I want to reshape it in PySpark so that all the keys (<code>foo</code>, <code>bar</code>, etc.) would become columns, namely:</p>
<pre><code>[Row(foo='1', bar='2', baz='aaa')]
</code></pre>
<p>Using <code>withColumn</code> works:</p>
<pre><code>(df
 .withColumn('foo', df.Parameters['foo'])
 .withColumn('bar', df.Parameters['bar'])
 .withColumn('baz', df.Parameters['baz'])
 .drop('Parameters')
).collect()
</code></pre>
<p>But <strong>I need a solution that doesn't explicitly mention the column names</strong>, as I have dozens of them.</p>