Spark Target log file already exists
<p>I am running a spark job. I can see that my job always fails and in the history server I see error</p>

<pre><code>java.io.IOException: Target log file already exists (hdfs://nameservice1/user/spark/applicationHistory/application_1456200816465_392022)
    at org.apache.spark.scheduler.EventLoggingListener.stop(EventLoggingListener.scala:201)
    at org.apache.spark.SparkContext$$anonfun$stop$5.apply(SparkContext.scala:1394)
    at org.apache.spark.SparkContext$$anonfun$stop$5.apply(SparkContext.scala:1394)
    at scala.Option.foreach(Option.scala:236)
    at org.apache.spark.SparkContext.stop(SparkContext.scala:1394)
    at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$3.run(ApplicationMaster.scala:107)
    at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54)
</code></pre>

<p>This is very confusing because it doesn't even enter my code.... I have no idea what is going wrong here. I tried deleting the directory and running again... but same issue.</p>