Spark on YARN resource manager: Relation between YARN Containers and Spark Executors
<p>I'm new to Spark on YARN and don't understand the relation between the YARN <code>Containers</code> and the Spark <code>Executors</code>. I tried out the following configuration, based on the results of the <code>yarn-utils.py</code> script, that can be used to find optimal cluster configuration.</p>

<p>The Hadoop cluster (HDP 2.4) I'm working on:</p>

<ul>
<li>1 Master Node:

<ul>
<li>CPU: 2 CPUs with 6 cores each = 12 cores</li>
<li>RAM: 64 GB</li>
<li>SSD: 2 x 512 GB</li>
</ul></li>
<li>5 Slave Nodes:

<ul>
<li>CPU: 2 CPUs with 6 cores each = 12 cores</li>
<li>RAM: 64 GB</li>
<li>HDD: 4 x 3 TB = 12 TB</li>
</ul></li>
<li>HBase is installed (this is one of the parameters for the script below)</li>
</ul>

<p>So I ran <code>python yarn-utils.py -c 12 -m 64 -d 4 -k True</code> (c=cores, m=memory, d=hdds, k=hbase-installed) and got the following result:</p>

<pre><code> Using cores=12 memory=64GB disks=4 hbase=True
 Profile: cores=12 memory=49152MB reserved=16GB usableMem=48GB disks=4
 Num Container=8
 Container Ram=6144MB
 Used Ram=48GB
 Unused Ram=16GB
 yarn.scheduler.minimum-allocation-mb=6144
 yarn.scheduler.maximum-allocation-mb=49152
 yarn.nodemanager.resource.memory-mb=49152
 mapreduce.map.memory.mb=6144
 mapreduce.map.java.opts=-Xmx4915m
 mapreduce.reduce.memory.mb=6144
 mapreduce.reduce.java.opts=-Xmx4915m
 yarn.app.mapreduce.am.resource.mb=6144
 yarn.app.mapreduce.am.command-opts=-Xmx4915m
 mapreduce.task.io.sort.mb=2457
</code></pre>

<p>These settings I made via the Ambari interface and restarted the cluster. The values also match roughly what I calculated manually before.</p>

<p>I have now problems</p>

<ul>
<li>to find the optimal settings for my <code>spark-submit</code> script

<ul>
<li>parameters <code>--num-executors</code>, <code>--executor-cores</code> &amp; <code>--executor-memory</code>.</li>
</ul></li>
<li>to get the relation between the YARN container and the Spark executors</li>
<li>to understand the hardware information in my Spark History UI (less memory shown as I set (when calculated to overall memory by multiplying with worker node amount))</li>
<li>to understand the concept of the <code>vcores</code> in YARN, here I couldn't find any useful examples yet</li>
</ul>

<p>However, I found this post <a href="https://stackoverflow.com/questions/14365218/what-is-a-container-in-yarn">What is a container in YARN?</a> , but this didn't really help as it doesn't describe the relation to the executors.</p>

<p>Can someone help to solve one or more of the questions?</p>