Create array of literals and columns from List of Strings in Spark
<p>I am trying to define functions in Scala that take a list of strings as input, and converts them into the columns passed to the dataframe array arguments used in the code below.</p>
<pre><code>val df = sc.parallelize(Array((1,1),(2,2),(3,3))).toDF(&quot;foo&quot;,&quot;bar&quot;)
val df2 = df
        .withColumn(&quot;columnArray&quot;,array(df(&quot;foo&quot;).cast(&quot;String&quot;),df(&quot;bar&quot;).cast(&quot;String&quot;)))
        .withColumn(&quot;litArray&quot;,array(lit(&quot;foo&quot;),lit(&quot;bar&quot;)))
</code></pre>
<p>More specifically, I would like to create functions <code>colFunction</code> and <code>litFunction</code> (or just one function if possible) that takes a list of strings as an input parameter and can be used as follows:</p>
<pre><code>val df = sc.parallelize(Array((1,1),(2,2),(3,3))).toDF(&quot;foo&quot;,&quot;bar&quot;)
val colString = List(&quot;foo&quot;,&quot;bar&quot;)
val df2 = df
         .withColumn(&quot;columnArray&quot;,array(colFunction(colString))
         .withColumn(&quot;litArray&quot;,array(litFunction(colString)))
</code></pre>
<p>I have tried mapping the <code>colString</code> to an Array of columns with all the transformations but this doesn't work.</p>