Possible causes of performance difference between two very similar Spark Dataframes
<p>I am working on improving the performance of some Spark operations for a recommendation engine I am prototyping. I have stumbled upon a significant performance differences between to DataFrames I am using. Below the results of describe() on both.</p>

<p>df1 (fast, numPartitions = 4):</p>

<pre><code>+-------+------------------+--------------------+
|summary|           item_id|          popularity|
+-------+------------------+--------------------+
|  count|            187824|              187824|
|   mean| 96693.34836868558|                 1.0|
| stddev|55558.023793621316|5.281958866780519...|
|    min|                 0|  0.9999999999999998|
|    max|            192806|                 1.0|
+-------+------------------+--------------------+
</code></pre>

<p>df2 (approx. 10x slower, numPartitions = ±170):</p>

<pre><code>+-------+-----------------+-----------------+
|summary|          item_id|            count|
+-------+-----------------+-----------------+
|  count|           187824|           187824|
|   mean|96693.34836868558|28.70869537439305|
| stddev|55558.02379362146|21.21976457710462|
|    min|                0|                1|
|    max|           192806|              482|
+-------+-----------------+-----------------+
</code></pre>

<p>Both DataFrames are cached, the same size in terms of rows (187824) and columns (2), and have an identical <code>item_id</code> column. The main difference is that frame 1 contains float in the second column, whereas frame 2 contains integer.</p>

<p>It seems as if every operation for DataFrame 2 is much slower, ranging from simply the <code>.describe().show()</code> operation, to more elaborate <code>.subtract().subtract().take()</code>. In the latter case DataFrame 2 takes 18 seconds, as opposed to 2 seconds for frame 1 (almost 10 times slower!).</p>

<p>I don't know where to begin to look for an explanation of the cause of this difference. Any tips or nudges in the right direction are greatly appreciated.</p>

<p><strong>UPDATE</strong>: as proposed by Viacheslav Rodionov the number of partitions of the dataframes seems to be the cause of the performance issues with df2.</p>

<p>Digging deeper, both dataframes are a result of <code>.groupBy().agg().sortBy()</code> operations on the same original dataframe. The <code>.groupBy().agg()</code> operation yields 200 partitions, and then <code>.sortBy()</code> returns respectively 4 and ±170 partitions, why could this be?</p>