Method for reducing memory load of Spark program
<p>I have a Spark program with calculates relations between users, i.e. it receives data set of type:</p>

<pre><code>RDD[(java.lang.Long, Map[(String, String), Integer])]
</code></pre>

<p>Where the Long is timestamp, and the map is a score relevant to tuples of two users. and should run some function over the scores and return the following type:</p>

<pre><code>Map[String, Map[java.lang.Long, java.lang.Double]]
</code></pre>

<p>Where the String is the first String in the tuple, and the map is the results of the function per timeslot. </p>

<p>In my case I have around 2000 users so the maps I receive are quite big (2000^2 per timeslot), and also the results relies on the previous timeslot results.</p>

<p>I am running the program locally and receiving <code>GC overhead limit exceeded</code>. I increased the heap memory to 14g using: <code>-Xmx14G</code> in vmarguments (I see the java process is occupying more than 12g of memory) but it didn't help.</p>

<p><strong>Currently implemented method</strong></p>

<p>I have tried several directions to decrease the memory consumption and currently came up with the following idea: since every timestamp relies only on the previous one I will collect every timeslot separately and keep the previous results on driver. In this manner I will run calculations only on part of the data and hopefully it will not crush the program.</p>

<p>The code:</p>

<pre><code>def calculateScorePerTimeslot(scorePerTimeslotRDD: RDD[(java.lang.Long, Map[(String, String), Integer])]): Map[String, Map[java.lang.Long, java.lang.Double]] = {
   var distancesPerTimeslotVarRDD = distancesPerTimeslotRDD.groupBy(_._1).sortBy(_._1)
   println("Start collecting all the results - cache the data!!")
   distancesPerTimeslotVarRDD.cache()
   println("Caching all the data has completed!")

   while(!distancesPerTimeslotVarRDD.isEmpty())
   {
     val dataForTimeslot: (java.lang.Long, Iterable[(java.lang.Long, Map[(String, String), Integer])]) = distancesPerTimeslotVarRDD.first()
     println("Retrieved data for timeslot: " + dataForTimeslot._1)

     //Code which is irrelevant for question - logic

     println("Removing timeslot: " + dataForTimeslot._1)
     distancesPerTimeslotVarRDD = distancesPerTimeslotVarRDD.filter(t =&gt; !t._1.equals(dataForTimeslot._1))
     println("Filtering has complete! - without: " + dataForTimeslot._1)
   }
}
</code></pre>

<p>Summary: Basically, the idea is to extract one timeslot at a time process it and save the results at driver - in this manner I try to reduce the size of data which passes on <code>collect</code>. </p>

<p><strong>Reason I write this post</strong></p>

<p>Unfortunately, this doesn't help me and the program still dies. My question is: is this manner of taking the first() item of a RDD and then filter it have the effect of iterating over the items on RDD? Are there other better ideas to tackle this kinds of question (better ideas which are not increasing the memory or moving to a real distributed cluster)? </p>