pySpark has a worker - driver version conflict when ran in Rodeo
<p>The following simple script works fine in pyspark when it is ran from the terminal:</p>

<pre><code>import pyspark

sc = pyspark.SparkContext()
foo = sc.parallelize([1,2])
foo.foreach(print)
</code></pre>

<p>But when ran in Rodeo, it produces an error, most important line of which says:</p>

<pre><code>Exception: Python in worker has different version 2.7 than that in driver 3.5, PySpark cannot run with different minor versions
</code></pre>

<p>And the full error output can be found at this link: <a href="http://pastebin.com/raw/unGuGLhq" rel="nofollow">http://pastebin.com/raw/unGuGLhq</a></p>

<p>My<code>$SPARK_HOME/conf/spark-env.sh</code> file contains the following lines:</p>

<pre><code>export PYSPARK_PYTHON=python3
export PYSPARK_DRIVER_PYTHON=python3
</code></pre>

<p>The problem persists despite that and putting the same lines in <code>~/.bashrc</code> doesn't solve the problem, either.</p>

<p>Rodeo version: 1.3.0</p>

<p>Spark version: 1.6.1</p>

<p>Platform: Linux</p>