How are stages split into tasks in Spark?
<p>Let's assume for the following that only one Spark job is running at every point in time.</p>

<h1>What I get so far</h1>

<p>Here is what I understand what happens in Spark:</p>

<ol>
<li>When a <code>SparkContext</code> is created, each worker node starts an executor.
Executors are separate processes (JVM), that connects back to the driver program. Each executor has the jar of the driver program. Quitting a driver, shuts down the executors. Each executor can hold some partitions.</li>
<li>When a job is executed, an execution plan is created according to the lineage graph.</li>
<li>The execution job is split into stages, where stages containing as many neighbouring (in the lineage graph) transformations and action, but no shuffles. Thus stages are separated by shuffles.</li>
</ol>

<p><a href="https://i.stack.imgur.com/NT1MB.png" rel="noreferrer"><img src="https://i.stack.imgur.com/NT1MB.png" alt="image 1"></a></p>

<p>I understand that</p>

<blockquote>
  <ul>
  <li>A task is a command sent from the driver to an executor by serializing the Function object.</li>
  <li>The executor deserializes (with the driver jar) the command (task) and executes it on a partition.</li>
  </ul>
</blockquote>

<p>but </p>

<h1>Question(s)</h1>

<p><em>How do I split the stage into those tasks?</em></p>

<p>Specifically:</p>

<ol>
<li>Are the tasks determined by the transformations and actions or can be multiple transformations/actions be in a task?</li>
<li>Are the tasks determined by the partition (e.g. one task per per stage per partition).</li>
<li>Are the tasks determined by the nodes (e.g. one task per stage per node)?</li>
</ol>

<h1>What I think (only partial answer, even if right)</h1>

<p>In <a href="https://0x0fff.com/spark-architecture-shuffle" rel="noreferrer">https://0x0fff.com/spark-architecture-shuffle</a>, the shuffle is explained with the image</p>

<p><a href="https://i.stack.imgur.com/KxhJE.png" rel="noreferrer"><img src="https://i.stack.imgur.com/KxhJE.png" alt="enter image description here"></a></p>

<p>and I get the impression that the rule is</p>

<blockquote>
  <p>each stage is split into #number-of-partitions tasks, with no regard for the number of nodes</p>
</blockquote>

<p>For my first image I'd say that I'd have 3 map tasks and 3 reduce tasks.</p>

<p>For the image from 0x0fff, I'd say there are 8 map tasks and 3 reduce tasks (assuming that there are only three orange and three dark green files).</p>

<h1>Open questions in any case</h1>

<p>Is that correct? But even if that is correct, my questions above are not all answered, because it is still open, whether multiple operations (e.g. multiple maps) are within one task or are separated into one tasks per operation.</p>

<h1>What others say</h1>

<p><a href="https://stackoverflow.com/questions/25276409/what-is-a-task-in-spark-how-does-the-spark-worker-execute-the-jar-file">What is a task in Spark? How does the Spark worker execute the jar file?</a> and <a href="https://stackoverflow.com/questions/28227941/how-does-the-apache-spark-scheduler-split-files-into-tasks">How does the Apache Spark scheduler split files into tasks?</a> are similar, but I did not feel that my question was answered clearly there.</p>