Spark: long delay between jobs
<p>So we are running spark job that extract data and do some expansive data conversion and writes to several different files.  Everything is running fine but I'm getting random expansive delays between resource intensive job finish and next job start.  </p>

<p>In below picture, we can see that job that was scheduled at 17:22:02 took 15 min to finish, which means I'm expecting next job to be scheduled around 17:37:02.  However, next job was scheduled at 22:05:59, which is +4 hours after job success.  </p>

<p>When I dig into next job's spark UI it show &lt;1 sec scheduler delay.  So I'm confused to where does this 4 hours long delay is coming from.  </p>

<p><a href="https://i.stack.imgur.com/1vwtD.png" rel="noreferrer"><img src="https://i.stack.imgur.com/1vwtD.png" alt="enter image description here"></a></p>

<p>(Spark 1.6.1 with Hadoop 2)</p>

<p><strong>Updated:</strong></p>

<p>I can confirm that David's answer below is spot on about how IO ops are handled in Spark is bit unexpected.  (It makes sense to that file write essentially does "collect" behind the curtain before it writes considering ordering and/or other operations.)  But I'm bit discomforted by the fact that I/O time is not included in job execution time.  I guess you can see it in "SQL" tab of spark UI as queries are still running even with all jobs being successful but you cannot dive into it at all.</p>

<p>I'm sure there are more ways to improve but below two methods were sufficient for me:</p>

<ol>
<li>reduce file count</li>
<li>set <code>parquet.enable.summary-metadata</code> to false</li>
</ol>