Building a row from a dict in pySpark
<p>I'm trying to dynamically build a row in pySpark 1.6.1, then build it into a dataframe.  The general idea is to extend the results of <code>describe</code> to include, for example, skew and kurtosis.  Here's what I thought should work:</p>

<pre><code>from pyspark.sql import Row

row_dict = {'C0': -1.1990072635132698,
            'C3': 0.12605772684660232,
            'C4': 0.5760856026559944,
            'C5': 0.1951877800894315,
            'C6': 24.72378589441825,
            'summary': 'kurtosis'}

new_row = Row(row_dict)
</code></pre>

<p>But this returns <code>TypeError: sequence item 0: expected string, dict found</code> which is a fairly clear error.  Then I found that if I defined the Row fields first, I <em>could</em> use a dict:</p>

<pre><code>r = Row('summary', 'C0', 'C3', 'C4', 'C5', 'C6')
r(row_dict)
&gt; Row(summary={'summary': 'kurtosis', 'C3': 0.12605772684660232, 'C0': -1.1990072635132698, 'C6': 24.72378589441825, 'C5': 0.1951877800894315, 'C4': 0.5760856026559944})
</code></pre>

<p>Which would be a fine step, except it doesn't seem like I can dynamically specify the fields in <code>Row</code>.  I need this to work for an unknown number of rows with unknown names.  According to the documentation you can actually go the other way:</p>

<pre><code>&gt;&gt;&gt; Row(name="Alice", age=11).asDict() == {'name': 'Alice', 'age': 11}
True
</code></pre>

<p>So it seems like I should be able to do this.  It also appears there may be some deprecated features from older versions that allowed this, for example <a href="https://spark.apache.org/docs/1.0.2/api/python/pyspark.sql.Row-class.html" rel="noreferrer">here</a>.  Is there a more current equivalent I'm missing?</p>