Spark Streaming: Issues when processing time > batch time
<p>I am running a spark streaming (1.6.1) on yarn using DirectAPI to read events from Kafka topic having 50 partitions and writing on HDFS. I have a batch interval of 60 seconds. I was receiving around 500K messages which was getting processed under 60 Sec.</p>

<p>Suddenly spark started receiving 15-20 million messages which took around 5-6 minutes to process with a batch interval of 60 seconds. I have configured <code>"spark.streaming.concurrentJobs=4"</code>.</p>

<p>So when batch takes a long time for processing spark initiate concurrent 4 active tasks to handle the backlog batches but still over a period of time batch backlog increases as batch interval is too less for such volume of data.</p>

<p>I have few doubts around this.</p>

<ol>
<li><p>When I start receiving 15-20 million messages &amp; time to process those messages is around 5-6 minutes with batch interval of 60 Sec. When I check my HDFS directory I see the files created for each 60 Sec with 50 part files, I am little confused here my batch is getting processed in 5-6 minutes, then how it is writing files on HDFS every 1 min &amp; <code>'saveAsTextFile'</code> action is called only once per batch. Total records from all the files 50 part files comes around 3.3 million.</p></li>
<li><p>In order to handle the processing of 15-20 million messages, I configured my batch interval to 8-10 minutes now spark started consuming around 35-40 million messages from Kafka &amp; again its processing time started exceeding batch interval.</p></li>
</ol>

<p>I have configured <code>'spark.streaming.kafka.maxRatePerPartition=50'</code> &amp; <code>'spark.streaming.backpressure.enabled=true'</code>.</p>