PySpark Row objects: accessing row elements by variable names
<p>One can access PySpark <code>Row</code> elements using the dot notation: given <code>r= Row(name="Alice", age=11)</code>, one can get the name or the age using <code>r.name</code> or <code>r.age</code> respectively. What happens when one needs to get an element whose name is stored in a variable <code>element</code>? One option is to do <code>r.toDict()[element]</code>. However, consider a situation where we have a large <code>DataFrame</code> and we wish to map a function on each row of that data frame. We can certainly do something like </p>

<pre><code>def f(row, element1, element2):
    row = row.asDict()
    return ", ".join(str(row[element1]), str(row[element2]))

result = dataframe.map(lambda row: f(row, 'age', 'name'))
</code></pre>

<p>However, it seems that calling <code>toDict()</code> on every row will be very inefficient. Is there a better way?</p>