Passing a data frame column and external list to udf under withColumn
<p>I have a Spark dataframe with the following structure. The bodyText_token has the tokens (processed/set of words). And I have a nested list of defined keywords</p>
<pre><code>root
 |-- id: string (nullable = true)
 |-- body: string (nullable = true)
 |-- bodyText_token: array (nullable = true)

keyword_list=[['union','workers','strike','pay','rally','free','immigration',],
['farmer','plants','fruits','workers'],['outside','field','party','clothes','fashions']]
</code></pre>
<p>I needed to check how many tokens fall under each keyword list and add the result as a new column of the existing dataframe.
Eg: if <code>tokens =[&quot;become&quot;, &quot;farmer&quot;,&quot;rally&quot;,&quot;workers&quot;,&quot;student&quot;]</code>
the result will be -&gt; <code>[1,2,0]</code></p>
<p>The following function worked as expected.</p>
<pre><code>def label_maker_topic(tokens,topic_words):
    twt_list = []
    for i in range(0, len(topic_words)):
        count = 0
        #print(topic_words[i])
        for tkn in tokens:
            if tkn in topic_words[i]:
                count += 1
        twt_list.append(count)
    
    return twt_list
</code></pre>
<p>I used udf under <code>withColumn</code> to access the function and I get an error. I think it's about passing an external list to a udf. Is there a way I can pass the external list and the dataframe column to a udf and add a new column to my dataframe?</p>
<pre><code>topicWord = udf(label_maker_topic,StringType())
myDF=myDF.withColumn(&quot;topic_word_count&quot;,topicWord(myDF.bodyText_token,keyword_list))
</code></pre>