Combine PySpark DataFrame ArrayType fields into single ArrayType field
<p>I have a PySpark DataFrame with 2 ArrayType fields:</p>

<pre><code>&gt;&gt;&gt;df
DataFrame[id: string, tokens: array&lt;string&gt;, bigrams: array&lt;string&gt;]
&gt;&gt;&gt;df.take(1)
[Row(id='ID1', tokens=['one', 'two', 'two'], bigrams=['one two', 'two two'])]
</code></pre>

<p>I would like to combine them into a single ArrayType field:</p>

<pre><code>&gt;&gt;&gt;df2
DataFrame[id: string, tokens_bigrams: array&lt;string&gt;]
&gt;&gt;&gt;df2.take(1)
[Row(id='ID1', tokens_bigrams=['one', 'two', 'two', 'one two', 'two two'])]
</code></pre>

<p>The syntax that works with strings does not seem to work here:</p>

<pre><code>df2 = df.withColumn('tokens_bigrams', df.tokens + df.bigrams)
</code></pre>

<p>Thanks!</p>