How can I pass extra parameters to UDFs in Spark SQL?
<p>I want to parse the date columns in a <code>DataFrame</code>, and for each date column, the resolution for the date may change (i.e. 2011/01/10 => 2011 /01 if the resolution is set to "Month"). </p>

<p>I wrote the following code:</p>

<pre><code>def convertDataFrame(dataframe: DataFrame, schema : Array[FieldDataType], resolution: Array[DateResolutionType]) : DataFrame =
{
  import org.apache.spark.sql.functions._
  val convertDateFunc = udf{(x:String, resolution: DateResolutionType) =&gt; SparkDateTimeConverter.convertDate(x, resolution)}
  val convertDateTimeFunc = udf{(x:String, resolution: DateResolutionType) =&gt; SparkDateTimeConverter.convertDateTime(x, resolution)}

  val allColNames = dataframe.columns
  val allCols = allColNames.map(name =&gt; dataframe.col(name))

  val mappedCols =
  {
    for(i &lt;- allCols.indices) yield
    {
      schema(i) match
      {
        case FieldDataType.Date =&gt; convertDateFunc(allCols(i), resolution(i)))
        case FieldDataType.DateTime =&gt; convertDateTimeFunc(allCols(i), resolution(i))
        case _ =&gt; allCols(i)
      }
    }
  }

  dataframe.select(mappedCols:_*)

}}
</code></pre>

<p>However it doesn't work. It seems that I can only pass <code>Column</code>s to UDFs. And I wonder if it will be very slow if I convert the <code>DataFrame</code> to <code>RDD</code> and apply the function on each row.</p>

<p>Does anyone know the correct solution? Thank you! </p>