Hadoop Yarn: How to limit dynamic self allocation of resources with Spark?
<p>In our Hadoop cluster that runs under Yarn we are having a problem that some "smarter" people are able to eat significantly larger chunks of resources by configuring Spark jobs in pySpark Jupyter notebooks like:</p>

<pre><code>conf = (SparkConf()
        .setAppName("name")
        .setMaster("yarn-client")
        .set("spark.executor.instances", "1000")
        .set("spark.executor.memory", "64g")
        )

sc = SparkContext(conf=conf)
</code></pre>

<p>This leads to the situation when these people literally squeeze out others less "smarter".</p>

<p>Is there a way to forbid users to self-allocate resources and leave resource allocation solely to Yarn?</p>