How to calculate the best numberOfPartitions for coalesce?
<p>So, I understand that in general one should use <code>coalesce()</code> when:</p>

<blockquote>
  <p>the number of partitions decreases due to a <code>filter</code> or some other operation that may result in reducing the original dataset (RDD, DF). <code>coalesce()</code> is useful for running operations more efficiently after filtering down a large dataset.</p>
</blockquote>

<p>I also understand that it is less expensive than <code>repartition</code> as it reduces shuffling by moving data only if necessary. My problem is how to define the parameter that <code>coalesce</code> takes (<code>idealPartionionNo</code>). I am working on a project which was passed to me from another engineer and he was using the below calculation to compute the value of that parameter. </p>

<pre><code>// DEFINE OPTIMAL PARTITION NUMBER
implicit val NO_OF_EXECUTOR_INSTANCES = sc.getConf.getInt("spark.executor.instances", 5)
implicit val NO_OF_EXECUTOR_CORES = sc.getConf.getInt("spark.executor.cores", 2)

val idealPartionionNo = NO_OF_EXECUTOR_INSTANCES * NO_OF_EXECUTOR_CORES * REPARTITION_FACTOR
</code></pre>

<p>This is then used with a <code>partitioner</code> object:</p>

<pre><code>val partitioner = new HashPartitioner(idealPartionionNo)
</code></pre>

<p>but also used with:</p>

<pre><code>RDD.filter(x=&gt;x._3&lt;30).coalesce(idealPartionionNo)
</code></pre>

<p>Is this the right approach? What is the main idea behind the <code>idealPartionionNo</code> value computation? What is the <code>REPARTITION_FACTOR</code>? How do I generally work to define that? </p>

<p>Also, since YARN is responsible for identifying the available executors on the fly is there a way of getting that number (<code>AVAILABLE_EXECUTOR_INSTANCES</code>) on the fly and use that for computing <code>idealPartionionNo</code> (i.e. replace <code>NO_OF_EXECUTOR_INSTANCES</code> with <code>AVAILABLE_EXECUTOR_INSTANCES</code>)?</p>

<p>Ideally, some actual examples of the form: </p>

<ul>
<li>Here 's a <em>dataset</em> (<em>size</em>); </li>
<li>Here's a number of transformations and possible reuses of an RDD/DF. </li>
<li>Here is where you should repartition/coalesce.  </li>
<li>Assume you have <code>n</code> <em>executors</em> with <code>m</code> <em>cores</em> and a <em>partition factor</em> equal to <code>k</code></li>
</ul>

<p>then:</p>

<ul>
<li>The ideal number of partitions would be ==> ??? </li>
</ul>

<p>Also, if you can refer me to a nice blog that explains these I would really appreciate it.</p>