How does Spark build its classpath?
<p>Could you share any knowledge on Exactly how spark build its classpath and determine which jars would be available on the driver/executors at runtime?</p>

<p>Here is what I can gather, so please let me know what I am missing or if I misunderstood. There doesn't seems to be a single authoritative documentation on the matter.</p>

<p>also ... the more important question is, does the <code>Thrift JDBC/ODBC server</code> use the same approach?</p>

<h3>Driver</h3>

<ul>
<li>value of <code>--jars</code> to <code>spark-submit</code></li>
<li><code>--packages</code> to <code>spark-submit</code> and <code>spark-shell</code></li>
<li>the app jar argument to <code>spark-submit</code> (the uber jar)</li>
<li><code>--driver-class-path</code> to <code>spark-submit</code></li>
<li><code>SPARK_HOME/lib</code></li>
<li><code>$HADOOP_CONF_DIR</code></li>
<li><code>$HADOOP_CLASSPATH</code> as determined by <code>$HADOOP_CONF_DIR/hadoop-env.sh</code></li>
<li>Any <code>spark.driver.extraClassPath</code> that find its way to <code>SparkConf</code></li>
</ul>

<h3>Executor</h3>

<ul>
<li>Same as the Driver?</li>
<li>Any <code>spark.executor.extraClassPath</code> that find its way to <code>SparkConf</code></li>
</ul>

<p>Am I missing anything? Is this consistent across <code>spark-submit</code> and the spark daemons? (such as the Thrift server or <code>spark-shell</code>).</p>