How does Spark manage stages?
<p>I am trying to understand how jobs and stages are defined in spark, and for that I am now using the code that I found <a href="https://i.stack.imgur.com/8EVbA.png" rel="noreferrer">here</a> and spark UI. In order to see it on spark UI I had to copy and paste the text on the files several times so it takes more time to process.</p>

<p>Here is the output of spark UI:</p>

<p><img src="https://i.stack.imgur.com/8EVbA.png" alt="Jobs"></p>

<p>Now, I understand that there are three jobs because there are three actions and also that the stages are generated because of shuffle actions, but what I don't understand is why in the Job 1 stages 4, 5 and 6 are the same as stages 0, 1 and 2 of Job 0 and the same happens for Job 2. </p>

<p>How can I know what stages will be in more than a job only seeing the java code (before executing anything)? And also, why are stage 4 and 9 skipped and how can I know it will happen before executing?</p>