Spark: executor memory exceeds physical limit
<p>My input dataset is about 150G.
I am setting</p>

<pre><code>--conf spark.cores.max=100 
--conf spark.executor.instances=20 
--conf spark.executor.memory=8G 
--conf spark.executor.cores=5 
--conf spark.driver.memory=4G
</code></pre>

<p>but since data is not evenly distributed across executors, I kept getting </p>

<pre><code>Container killed by YARN for exceeding memory limits. 9.0 GB of 9 GB physical memory used
</code></pre>

<p>here are my questions:</p>

<pre><code>1. Did I not set up enough memory in the first place? I think 20 * 8G &gt; 150G, but it's hard to make perfect distribution, so some executors will suffer
2. I think about repartition the input dataFrame, so how can I determine how many partition to set? the higher the better, or?
3. The error says "9 GB physical memory used", but i only set 8G to executor memory, where does the extra 1G come from?
</code></pre>

<p>Thank you!</p>