spark Container killed by YARN for exceeding memory limits
<p>I'm running spark in AWS EMR.
it's a simple computation of PageRank, dataset is 8GB</p>

<p>I use 6 m3.xlarge for my cluster,each 16GB memory</p>

<p>here is the configuration</p>

<pre><code>spark.executor.instances   4
spark.executor.cores       8
spark.driver.memory    10473M
spark.executor.memory   9658M
</code></pre>

<p>I'm new to Spark, so I really have no common sense about how memory-consuming it's......Is my cluster too small?  And is the memory a hard limit? or , small memory is ok but just computing slow....</p>

<p>here is my code (in our homework we're not allowed to use GraphX, so I just use plain function)</p>

<pre><code>import org.apache.spark.SparkConf;
import org.apache.spark.SparkContext;
import org.apache.spark.rdd.RDD;

object PageRank {
  def main(args: Array[String]): Unit = {
    val conf = new SparkConf().setAppName("pageRank")

    val sc = new SparkContext(conf)

    val iterNum = 10
    val file = sc.textFile("hdfs:///input")


    val all = file.flatMap { line =&gt; line.split("\t") }.distinct()
    val contributor = file.map ( line =&gt; line.split("\t")(0)).distinct()
    val dangling = all.subtract(contributor)

    val graphDangling = dangling.cartesian(all).groupByKey()

    val graph0 = file.map(line =&gt; { val temp = line.split("\t"); (temp(0), temp(1)) }).distinct().groupByKey()
    val graph = graph0.union(graphDangling)
    graph.cache()

    var ranks = graph.mapValues { x =&gt; 1.0 }

    var i = 0

    for (i &lt;- 0 to iterNum) {
      val contriReceive = graph.join(ranks).values.flatMap {
        case (followees, rank) =&gt;
          {
            val size = followees.size;
            followees.map(followee =&gt; (followee, rank / size))
          }
      }
      ranks = contriReceive.reduceByKey(_ + _).mapValues { x =&gt; 0.15 + 0.85 * x }
    }

    val result = ranks.map{case(user,rank)=&gt;user+"\t"+rank}

    ranks.saveAsTextFile("hdfs:///pagerank-output")

    sc.stop()
  }

}
</code></pre>

<p>I'm not very sure about the RDD and memory management. There are many intermediate RDD, do i have to explicitly release them to free the resource? if so, how to ....just assign to null and gc will deal with it?</p>