Can a PySpark Kernel(JupyterHub) run in yarn-client mode?
<p>My Current Setup:</p>

<ul>
<li>Spark EC2 Cluster with HDFS and YARN</li>
<li>JuputerHub(0.7.0)</li>
<li>PySpark Kernel with python27</li>
</ul>

<p>The very simple code that I am using for this question:</p>

<pre><code>rdd = sc.parallelize([1, 2])
rdd.collect()
</code></pre>

<p>The PySpark kernel that <strong>works as expected</strong> in Spark standalone has the following environment variable in the kernel json file:</p>

<pre><code>"PYSPARK_SUBMIT_ARGS": "--master spark://&lt;spark_master&gt;:7077 pyspark-shell"
</code></pre>

<p>However, when I try to run in yarn-client mode it is getting stuck forever, while the log output from the JupyerHub logs is:</p>

<pre><code>16/12/12 16:45:21 WARN YarnScheduler: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
16/12/12 16:45:36 WARN YarnScheduler: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
16/12/12 16:45:51 WARN YarnScheduler: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
16/12/12 16:46:06 WARN YarnScheduler: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
</code></pre>

<p>As described <a href="http://spark.apache.org/docs/latest/running-on-yarn.html" rel="nofollow noreferrer">here</a> I have added the <strong>HADOOP_CONF_DIR</strong> env. variable to point to the directory where the Hadoop configurations are, and changed <strong>PYSPARK_SUBMIT_ARGS</strong> <code>--master</code> property to "<strong>yarn-client</strong>". Also i can confirm that there are no other jobs running during this and that the workers are correctly registered.</p>

<p>I am under the impression that it is possible to configure a JupyterHub Notebook with a PySpark kernel to run with YARN as <a href="https://github.com/jupyterhub/jupyterhub/issues/438" rel="nofollow noreferrer">other people have done it</a>, if this indeed is the case what I am I doing wrong?</p>