Spark udf with non column parameters
<p>I want to pass a variable and not a column to a UDF in spark.</p>

<p>The map is of the following format
<a href="https://stackoverflow.com/questions/41238117/spark-dataframe-to-nested-map">Spark dataframe to nested map</a></p>

<pre><code>val joinUDF = udf((replacementLookup: Map[String, Double], newValue: String) =&gt; {
    replacementLookup.get(newValue) match {
      case Some(tt) =&gt; tt
      case None =&gt; 0.0
    }
  })
</code></pre>

<p>Should be mapped like</p>

<pre><code>(columnsMap).foldLeft(df) {
    (currentDF, colName) =&gt;
      {
        println(colName._1)
        println(colName._2)
        currentDF
          .withColumn("myColumn_" + colName._1, joinUDF(colName._2, col(colName._1)))
      }
  }
</code></pre>

<p>But throws </p>

<pre><code>type mismatch;
[error]  found   : Map
[error]  required: org.apache.spark.sql.Column
[error]           .withColumn("myColumn_" + colName._1, joinUDF(colName._2, col(colName._1)))
</code></pre>