Spark on YARN: Less executor memory than set via spark-submit
<p>I'm using Spark in a YARN cluster (HDP 2.4) with the following settings:</p>

<ul>
<li>1 Masternode

<ul>
<li>64 GB RAM (48 GB usable)</li>
<li>12 cores (8 cores usable)</li>
</ul></li>
<li>5 Slavenodes

<ul>
<li>64 GB RAM (48 GB usable) each</li>
<li>12 cores (8 cores usable) each </li>
</ul></li>
<li>YARN settings

<ul>
<li>memory of all containers (of one host): 48 GB</li>
<li>minimum container size = maximum container size = 6 GB</li>
<li>vcores in cluster = 40 (5 x 8 cores of workers)</li>
<li>minimum #vcores/container = maximum #vcores/container = 1</li>
</ul></li>
</ul>

<p>When I run my spark application with the command <code>spark-submit --num-executors 10 --executor-cores 1 --executor-memory 5g ...</code> Spark should give each executor 5 GB of RAM right (I set memory only to 5g due to some overhead memory of ~10%).</p>

<p>But when I had a look in the Spark UI, I saw that each executor only has 3.4 GB of memory, see screenshot: </p>

<p><a href="https://i.stack.imgur.com/psvrP.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/psvrP.png" alt="screenshot"></a></p>

<p>Can someone explain why there's so less memory allocated?</p>