PYSPARK_PYTHON works with --deploy-mode client but not --deploy-mode cluster
<p>I'm trying to run a python script using a custom python and deploy <code>--deploy-mode cluster</code> on an Enterprise 4.2 cluster.  </p>



<pre class="lang-sh prettyprint-override"><code>[biadmin@bi4c-xxxxx-mastermanager ~]$ hive
hive&gt; CREATE TABLE pokes (foo INT, bar STRING);
OK
Time taken: 2.147 seconds
hive&gt; LOAD DATA LOCAL INPATH '/usr/iop/4.2.0.0/hive/doc/examples/files/kv1.txt' OVERWRITE INTO TABLE pokes;
Loading data to table default.pokes
Table default.pokes stats: [numFiles=1, numRows=0, totalSize=5812, rawDataSize=0]
OK
Time taken: 0.49 seconds
hive&gt; 
</code></pre>

<p>Then I create a simple pyspark script:</p>



<pre class="lang-py prettyprint-override"><code>[biadmin@bi4c-xxxxxx-mastermanager ~]$ cat test_pokes.py
from pyspark import SparkContext
from pyspark.sql import HiveContext

sc = SparkContext()
hc = HiveContext(sc)

pokesRdd = hc.sql('select * from pokes')
print( pokesRdd.collect() )
</code></pre>

<p>I then try to run the script like this:</p>



<pre class="lang-sh prettyprint-override"><code>export PYSPARK_PYTHON=/home/biadmin/anaconda2/bin/python2.7 
export PYSPARK_DRIVER_PYTHON=/home/biadmin/anaconda2/bin/python2.7
spark-submit --master yarn \
             --deploy-mode cluster \
             --jars /usr/iop/4.2.0.0/hive/lib/datanucleus-api-jdo-3.2.6.jar,/usr/iop/4.2.0.0/hive/lib/datanucleus-core-3.2.10.jar,/usr/iop/4.2.0.0/hive/lib/datanucleus-rdbms-3.2.9.jar \
             --files /usr/iop/current/spark-client/conf/hive-site.xml \ 
             test_pokes.py
</code></pre>

<p>This runs on the yarn cluster but does NOT use the <code>PYSPARK_PYTHON</code> variable.</p>

<p>However, if I use <code>--deploy-mode client</code>, <code>PYSPARK_PYTHON</code> is used ok.</p>

<hr>

<p><strong>Update</strong></p>

<p>I have tried adding this before the SparkContext is initialised:</p>

<pre class="lang-sh prettyprint-override"><code>os.environ["PYSPARK_PYTHON"] = '/home/biadmin/anaconda2/bin/python2.7'
os.environ["PYSPARK_DRIVER_PYTHON"] = '/home/biadmin/anaconda2/bin/python2.7'
</code></pre>

<p>Also, tried setting <code>--conf spark.yarn.appMasterEnv.PYSPARK_PYTHON=/home/biadmin/anacond‌​a2/bin/python2.7</code></p>

<pre class="lang-sh prettyprint-override"><code>Container: container_e09_1477084339086_0508_02_000001 on bi4c-xxxxxx-data-1.bi.services.bluemix.net_45454
==========================================================================================================
LogType:stderr
...
java.io.IOException: Cannot run program "/home/biadmin/anaconda2/bin/python2.7": error=2, No such file or directory
</code></pre>

<p>However, </p>

<pre class="lang-sh prettyprint-override"><code>[biadmin@bi4c-xxxxxx-mastermanager ~]$ ssh bi4c-xxxxxx-data-1.bi.services.bluemix.net
[biadmin@bi4c-xxxxxx-data-2 ~]$ ls /home/biadmin/anaconda2/bin/python2.7
/home/biadmin/anaconda2/bin/python2.7
</code></pre>