Saving CSV file with partitionBy in Spark
<p>I'm trying to save a dataframe as CSV file partitioned by a column.</p>

<pre><code>val schema = new StructType(
      Array(
        StructField("ID",IntegerType,true),
        StructField("State",StringType,true),
        StructField("Age",IntegerType,true)
      )
)

val df = sqlContext.read.format("com.databricks.spark.csv")
        .options(Map("path" -&gt; filePath).schema(schema).load()

df.write.partitionBy("State").format("com.databricks.spark.csv").save(outputPath)
</code></pre>

<p>But the output is not saved with any partition info. It looks like partitionBy was completely ignored. There were no errors. It works if I try the same with parquet format.</p>

<pre><code>df.write.partitionBy("State").parquet(outputPath)
</code></pre>

<p>What am I missing here?</p>