Understanding Spark Streaming WAL failure guarantees
<p>I am trying to understand what will happen when Spark has an exception during processing, especially while streaming.</p>

<p>If I have a small code spinet like this:</p>

<pre class="lang-scala prettyprint-override"><code>myDStream.foreachRDD { (rdd: RDD[String]) =&gt;
  println(s"processed =&gt; [${rdd.collect().toList}]")
  throw new Exception("User exception...")
}
</code></pre>

<p>If I run this I will get output like this:</p>

<pre><code>[info] processed =&gt; [List(Item1)]
[error] 28-01-2016 17:41:18 ERROR JobScheduler:96 - Error running job streaming job 1453999278000 ms.0
[error] java.lang.Exception: User exception...
...
[info] processed =&gt; [List(Item2)]
[error] 28-01-2016 17:41:19 ERROR JobScheduler:96 - Error running job streaming job 1453999279000 ms.0
[error] java.lang.Exception: User exception...
</code></pre>

<p>First "Item1" is processed, and it fails (of course).  In the next batch "Item2" is processed.  The record "Item1" has now been lost.</p>

<p>If I change my code so that the exception occurs inside a task:</p>

<pre class="lang-scala prettyprint-override"><code>myDStream.foreachRDD { (rdd: RDD[String]) =&gt;
  println(s"processed =&gt; [${rdd.collect().toList}]")
  rdd.map{case x =&gt; throw new Exception("User exception...") }.collect()
}
</code></pre>

<p>Then the map closure will be retried, but once it has failed enough times the record is discarded and processing continues to the next record.</p>

<p>Is it possible to ensure that records are not discarded, even if this means stopping the application?  I have the WAL enabled.</p>

<p><strong>Update:</strong> Tathagata Das responded on the Spark User Mailing List <a href="https://www.mail-archive.com/user@spark.apache.org/msg44910.html" rel="nofollow">here</a>.</p>