Pyspark append executor environment variable
<p>Is it possible to append a value to the PYTHONPATH of a worker in spark?</p>
<p>I know it is possible to go to each worker node, configure spark-env.sh file and do it, but I want a more flexible approach</p>
<p>I am trying to use setExecutorEnv method, but with no success</p>
<pre><code>conf = SparkConf().setMaster(&quot;spark://192.168.10.11:7077&quot;)\
              .setAppName('myname')\
              .set(&quot;spark.cassandra.connection.host&quot;, &quot;192.168.10.11&quot;) /
              .setExecutorEnv('PYTHONPATH', '$PYTHONPATH:/custom_dir_that_I_want_to_append/')
</code></pre>
<p>It creates a pythonpath env.variable on each executor, force it to be lower_case, and does not interprets $PYTHONPATH command to append the value.</p>
<p>I end up with two different env.variables,</p>
<pre><code>pythonpath  :  $PYTHONPATH:/custom_dir_that_I_want_to_append
PYTHONPATH  :  /old/path/to_python
</code></pre>
<p>The first one is dynamically created and the second one already existed before.</p>
<p>Does anyone know how to do it?</p>