Spark 1.6 DirectFileOutputCommitter
<p>I am having a problem saving text files to S3  using pyspark. I am able to save to S3, but it first uploads to a _temporary on S3 and then proceeds to copy to the intended location. This increases the jobs run time significantly. I have attempted to compile a DirectFileOutputComitter which should write directly to the intended S3 url, but I cannot get Spark to utilize this class.</p>

<p>Example: </p>

<blockquote>
  <p>someRDD.saveAsTextFile("s3a://somebucket/savefolder")</p>
</blockquote>

<p>this creates a </p>

<blockquote>
  <p>s3a://somebucket/savefolder/_temporary/</p>
</blockquote>

<p>directory which is then written to after which a S3 copy operation moves the files back to </p>

<blockquote>
  <p>s3a://somebucket/savefolder</p>
</blockquote>

<p>My question is does anyone have a working jar of the DirectFileOutputCommiter, or if anyone has experience working around this issue. </p>

<p>Relevant Links:</p>

<ol>
<li><a href="https://issues.apache.org/jira/browse/HADOOP-10400" rel="nofollow">https://issues.apache.org/jira/browse/HADOOP-10400</a></li>
<li><a href="https://gist.github.com/aarondav/c513916e72101bbe14ec" rel="nofollow">https://gist.github.com/aarondav/c513916e72101bbe14ec</a></li>
<li><a href="https://mail-archives.apache.org/mod_mbox/spark-user/201503.mbox/%3C029201d06334" rel="nofollow">https://mail-archives.apache.org/mod_mbox/spark-user/201503.mbox/%3C029201d06334</a>$a0871180$e1953480$@gmail.com%3E</li>
<li><a href="http://tech.grammarly.com/blog/posts/Petabyte-Scale-Text-Processing-with-Spark.html" rel="nofollow">http://tech.grammarly.com/blog/posts/Petabyte-Scale-Text-Processing-with-Spark.html</a></li>
</ol>