`HIVE_METASTORE_JARS` variable ignored by PySpark
<p>I'm running PySpark 1.6.0 against Hive 0.14. The config option <code>spark.sql.hive.metastore.version</code> is set to 0.14 but <code>spark.sql.hive.metastore.jars</code> is not set, so PySpark is trying to use the built in and failing to access Hive with the message</p>

<pre><code>IllegalArgumentException: u'Builtin jars can only be used when hive execution version == hive metastore version. Execution: 1.2.1 != Metastore: 0.14. Specify a vaild path to the correct hive jars using $HIVE_METASTORE_JARS or change spark.sql.hive.metastore.version to 1.2.1.'
</code></pre>

<p>as expected. I set on the shell the variable <code>HIVE_METASTORE_JARS</code> to the correct folder for the Hive jars for 0.14 but PySpark ignores it. Passing it with <code>--jars hive-metastore.jar</code> does not seem to work (it hangs completely).</p>

<p>Is there a way to make PySpark see these jars without changing the <code>spark.sql.hive.metastore.jars</code> variable on <code>spark-conf</code>? </p>