Spark skewing data to few executors
<p>I'm running spark on standalone mode with 21 executors, and when I load my first SQL table using my sqlContext, I partition it in a way such that the data is perfectly distributed among all blocks by partitioning on a column that is sequential integers:</p>

<pre><code>val brDF = sqlContext.load("jdbc", Map("url" -&gt; srcurl, "dbtable" -&gt; "basereading", "partitionColumn" -&gt; "timeperiod", "lowerBound" -&gt;"2", "upperBound" -&gt; "35037", "numPartitions" -&gt; "100"))
</code></pre>

<p>Additionally, the blocks are nicely distributed on each cluster so that each cluster has a similar memory usage. 
Unfortunately, when I join it with a much smaller table idoM like so:</p>

<pre><code>val mrDF = idoM.as('idom).join(brS1DF.as('br), $"idom.idoid" === $"br.meter")
</code></pre>

<p>Where idoM is a 1 column table and cache the result, the distribution of the way the RDD blocks are stored on the cluster changes: </p>

<p><a href="http://i.stack.imgur.com/vgdwE.jpg" rel="nofollow">screenshot of spark UI executors sorted by number of RDD blocks</a></p>

<p>Now, there are suddenly more RDD blocks on my fourth cluster and it uses more memory. Upon checking each RDD, their blocks seem to still be distributed nicely so my partitioning is still fine, just that all the blocks seem to only want to be written on one cluster, defeating the purpose of having multiple to begin with. </p>

<p>I suspect that my problem has something similar to 
<a href="http://apache-spark-user-list.1001560.n3.nabble.com/RDD-Blocks-skewing-to-just-few-executors-tc19112.html" rel="nofollow">this question on the Apache mail list</a>
but there is no answer, so anything would be greatly appreciated.</p>