Adding custom jars to pyspark in jupyter notebook
<p>I am using the <strong>Jupyter notebook with Pyspark</strong> with the following <strong>docker image</strong>: <a href="https://github.com/jupyter/docker-stacks/tree/master/all-spark-notebook" rel="noreferrer">Jupyter all-spark-notebook</a></p>

<p>Now I would like to write a <strong>pyspark streaming application which consumes messages from Kafka</strong>. In the <a href="http://spark.apache.org/docs/latest/streaming-kafka-integration.html" rel="noreferrer">Spark-Kafka Integration guide</a> they describe how to deploy such an application using spark-submit (it requires linking an external jar - explanation is in <em>3. Deploying</em>). But since I am using Jupyter notebook I never actually run the <code>spark-submit</code> command, I assume it gets run in the back if I press execute.</p>

<p>In the <code>spark-submit</code> command you can specify some parameters, one of them is <code>-jars</code>, but it is not clear to me how I can set this parameter from the notebook (or externally via environment variables?). I am assuming I can link this external jar dynamically via the <code>SparkConf</code> or the <code>SparkContext</code> object. Has anyone experience on how to perform the linking properly from the notebook?</p>