pyspark 1.6.0 write to parquet gives "path exists" error
<p>I read in from parquet file(s) from different folders, take e.g. February this year (one folder = one day)</p>

<pre><code>indata = sqlContext.read.parquet('/data/myfolder/201602*')
</code></pre>

<p>do some very simple grouping and aggregation</p>

<pre><code>outdata = indata.groupby(...).agg()
</code></pre>

<p>and want to store again. </p>

<pre><code>outdata.write.parquet(outloc)
</code></pre>

<p>Here is how I run the script from bash:</p>

<pre><code>spark-submit 
  --master yarn-cluster 
  --num-executors 16 
  --executor-cores 4 
  --driver-memory 8g
  --executor-memory 16g 
  --files /etc/hive/conf/hive-site.xml  
  --driver-java-options 
  -XX:MaxPermSize=512m 
  spark_script.py
</code></pre>

<p>This generates multiple jobs (is that the right term?). First job runs successfully. Subsequent jobs fail with the following error:</p>

<pre><code>  Traceback (most recent call last):
  File "spark_generate_maps.py", line 184, in &lt;module&gt;
    outdata.write.parquet(outloc)
  File "/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/spark/python/lib/pyspark.zip/pyspark/sql/readwriter.py", line 471, in parquet
  File "/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/spark/python/lib/py4j-0.9-src.zip/py4j/java_gateway.py", line 813, in __call__
  File "/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/spark/python/lib/pyspark.zip/pyspark/sql/utils.py", line 51, in deco
pyspark.sql.utils.AnalysisException: u'path OBFUSCATED_PATH_THAT_I_CLEANED_BEFORE_SUBMIT already exists.;'
</code></pre>

<p><strong>When I give only one folder as input, this works fine.</strong></p>

<p><strong>So it seems the first job creates the folder, all subsequent jobs fail to write into that folder. Why?</strong></p>

<p>just in case this could help anybody:</p>

<p><strong>imports:</strong></p>

<pre><code>from pyspark import SparkContext, SparkConf, SQLContext
from pyspark.sql import HiveContext
from pyspark.sql.types import *
from pyspark.sql.functions import udf, collect_list, countDistinct, count
import pyspark.sql.functions as func
from pyspark.sql.functions import lit
import numpy as np
import sys
import math
</code></pre>

<p><strong>config:</strong></p>

<pre><code>conf = SparkConf().setAppName('spark-compute-maps').setMaster('yarn-cluster')
sc = SparkContext(conf=conf)
sqlContext = HiveContext(sc)
</code></pre>