Apache Spark: setting executor instances
<p>I run my Spark application on YARN with parameters:</p>

<p>in spark-defaults.conf:</p>

<pre><code>spark.master yarn-client
spark.driver.cores 1
spark.driver.memory 1g
spark.executor.instances 6
spark.executor.memory 1g
</code></pre>

<p>in yarn-site.xml:</p>

<pre><code>yarn.nodemanager.resource.memory-mb 10240
</code></pre>

<p>All other parameters are set to default.</p>

<p>I have a 6-node cluster and the Spark Client component is installed on each node. 
Every time I run the application there are only 2 executors and 1 driver visible in the Spark UI. Executors appears on different nodes.</p>

<p>Why can't Spark create more executors? Why are only 2 instead of 6?</p>

<p>I found a very similar question: <a href="https://stackoverflow.com/questions/29940711/apache-spark-setting-executor-instances-does-not-change-the-executors">Apache Spark: setting executor instances does not change the executors</a>, but increasing the memoty-mb parameter didn't help in my case.</p>