Specify format of Timestamp written by pyspark
<p>Background is a simply pyspark programme that I developed on 1.6 using databricks csv read/writer, and all was happy. My dataframe had a timestamp column, which was written out in a standard <code>YYYY-MM-DD HH24:MI:SS</code> format. </p>

<pre><code>foo,bar,2016-10-14 14:30:31.985 
</code></pre>

<p>Now I’m running it on EMR with Spark 2, and the timestamp column is being written as an <strong>epoch in microseconds</strong>. This causes a problem because the target (Redshift) can’t natively handle this (only seconds or milliseconds).</p>

<pre><code>foo,bar,1476455559456000
</code></pre>

<p>Looking at <a href="http://spark.apache.org/docs/latest/api/python/pyspark.sql.html#pyspark.sql.DataFrameWriter.csv" rel="nofollow">the docs</a>, it seems I should be able to specify the format used with <code>timestampFormat</code>, but I just get an error : </p>

<pre><code>TypeError: csv() got an unexpected keyword argument 'timestampFormat'
</code></pre>

<p>Am I calling this wrong, or does the option not exist? Any other way to cleanly get my timestamp data out in a format that's <em>not</em> microseconds (milli would be fine, or any other standard time format really)</p>

<hr>

<p>Simple code to reproduce: </p>

<pre><code>df = sqlContext.createDataFrame([('foo','bar')]).withColumn('foo',pyspark.sql.functions.current_timestamp())
df.printSchema()
df.show()

# Use the new Spark 2 native method
df.write.csv(path='/tmp/foo',mode='overwrite')

# Use the databricks CSV method, pre Spark 2
df.write.save(path='/tmp/foo2',format='com.databricks.spark.csv',mode='overwrite')
</code></pre>