Iteratively process with RDD can't last long
<p>I need to join 2 RDDs which are too big to load and join in a single processing. So I fetch some records from source RDD and destination RDD and join them iteratively. But I found with time went by the join speed became slower and slower and at last the program stopped at a certain stage. The stage's status was staging and never changed. It seems that the RDDs declared in the iteration had not been freed and thus the system has no enough memory for new RDDs. How to fix it?</p>

<pre><code>var A: RDD[(Int, UUID)] = …
var B: RDD[(Int, UUID)] = …
for (i &lt;- 0 until 64) {
    var tmpA = A.filter(x =&gt; x._1%64 == i)
    var tmpB = B.filter(x =&gt; x._1%64 == i)
    var C = A.join(B)
    println(C.count)
}
</code></pre>