Scala: Spark SQL to_date(unix_timestamp) returning NULL
<p><code>Spark Version: spark-2.0.1-bin-hadoop2.7
Scala: 2.11.8</code></p>

<p>I am loading a raw csv into a DataFrame. In csv, although the column is support to be in date format, they are written as 20161025 instead of 2016-10-25. The parameter <code>date_format</code> includes string of column names that need to be converted to yyyy-mm-dd format.</p>

<p>In the following code, I first loaded the csv of Date column as StringType via the <code>schema</code>, and then I check if the <code>date_format</code> is not empty, that is there are columns that need to be converted to <code>Date</code> from <code>String</code>, then cast each column using <code>unix_timestamp</code> and <code>to_date</code>. However, in the <code>csv_df.show()</code>, the returned rows are all <code>null</code>.  </p>

<pre><code>def read_csv(csv_source:String, delimiter:String, is_first_line_header:Boolean, 
    schema:StructType, date_format:List[String]): DataFrame = {
    println("|||| Reading CSV Input ||||")

    var csv_df = sqlContext.read
        .format("com.databricks.spark.csv")
        .schema(schema)
        .option("header", is_first_line_header)
        .option("delimiter", delimiter)
        .load(csv_source)
    println("|||| Successfully read CSV. Number of rows -&gt; " + csv_df.count() + " ||||")
    if(date_format.length &gt; 0) {
        for (i &lt;- 0 until date_format.length) {
            csv_df = csv_df.select(to_date(unix_timestamp(
                csv_df(date_format(i)), "yyyy-­MM-­dd").cast("timestamp")))
            csv_df.show()
        }
    }
    csv_df
}
</code></pre>

<p>Returned Top 20 rows:</p>

<pre><code>+-------------------------------------------------------------------------+
|to_date(CAST(unix_timestamp(prom_price_date, YYYY-­MM-­DD) AS TIMESTAMP))|
+-------------------------------------------------------------------------+
|                                                                     null|
|                                                                     null|
|                                                                     null|
|                                                                     null|
|                                                                     null|
|                                                                     null|
|                                                                     null|
|                                                                     null|
|                                                                     null|
|                                                                     null|
|                                                                     null|
|                                                                     null|
|                                                                     null|
|                                                                     null|
|                                                                     null|
|                                                                     null|
|                                                                     null|
|                                                                     null|
|                                                                     null|
|                                                                     null|
+-------------------------------------------------------------------------+
</code></pre>

<p>Why am I getting all <code>null</code>?</p>