Spark dataframe saveAsTable vs save
<p>I am using spark 1.6.1 and I am trying to save a dataframe to an orc format.</p>

<p>The problem I am facing is that the save method is very slow, and it takes about 6 minutes for 50M orc file on each executor.
This is how I am saving the dataframe</p>

<pre><code>dt.write.format("orc").mode("append").partitionBy("dt").save(path)
</code></pre>

<p>I tried using saveAsTable to an hive table which is also using orc formats, and that seems to be faster about 20% to 50% faster, but this method has its own problems - it seems that when a task fails, retries will always fail due to file already exist.
This is how I am saving the dataframe</p>

<pre><code>dt.write.format("orc").mode("append").partitionBy("dt").saveAsTable(tableName)
</code></pre>

<p>Is there a reason save method is so slow?
Am I doing something wrong?</p>