Determining optimal number of Spark partitions based on workers, cores and DataFrame size
<p>There are several similar-yet-different concepts in Spark-land surrounding how work gets farmed out to different nodes and executed concurrently. Specifically, there is:</p>

<ul>
<li>The Spark Driver node (<code>sparkDriverCount</code>)</li>
<li>The number of worker nodes available to a Spark cluster (<code>numWorkerNodes</code>)</li>
<li>The number of Spark executors (<code>numExecutors</code>)</li>
<li>The DataFrame being operated on by all workers/executors, concurrently (<code>dataFrame</code>)</li>
<li>The number of rows in the <code>dataFrame</code> (<code>numDFRows</code>)</li>
<li>The number of partitions on the <code>dataFrame</code> (<code>numPartitions</code>)</li>
<li>And finally, the number of CPU cores available on each worker nodes (<code>numCpuCoresPerWorker</code>)</li>
</ul>

<p>I <em>believe</em> that all Spark clusters have <em>one-and-only-one</em> Spark Driver, and then 0+ worker nodes. If I'm wrong about that, please begin by correcting me! Assuming I'm more or less correct about that, let's lock in a few variables here. Let's say we have a Spark cluster with 1 Driver and 4 Worker nodes, and each Worker Node has 4 CPU cores on it (so a total of 16 CPU cores). So the "given" here is:</p>

<pre><code>sparkDriverCount = 1
numWorkerNodes = 4
numCpuCores = numWorkerNodes * numCpuCoresPerWorker = 4 * 4 = 16
</code></pre>

<p>Given that as the setup, I'm wondering how to determine a few things. Specifically:</p>

<ul>
<li>What is the relationship between <code>numWorkerNodes</code> and <code>numExecutors</code>? Is there some known/generally-accepted ratio of workers to executors? Is there a way to determine <code>numExecutors</code> given <code>numWorkerNodes</code> (or any other inputs)?</li>
<li>Is there a known/generally-accepted/optimal ratio of <code>numDFRows</code> to <code>numPartitions</code>? How does one calculate the 'optimal' number of partitions based on the size of the <code>dataFrame</code>?</li>
<li>I've heard from other engineers that a general 'rule of thumb' is: <code>numPartitions = numWorkerNodes * numCpuCoresPerWorker</code>, any truth to that? In other words, it prescribes that one should have 1 partition per CPU core.</li>
</ul>