import pymongo_spark doesn't work when executing with spark-commit
<p>I am running into problem running my script with spark-submit. The main script won't even run because <code>import pymongo_spark</code> returns <code>ImportError: No module named pymongo_spark</code></p>

<p>I checked <a href="https://stackoverflow.com/questions/33391840/getting-spark-python-and-mongodb-to-work-together">this thread</a> and <a href="https://stackoverflow.com/questions/28485851/issues-setting-up-apache-spark-with-mongo-hadoop-connector">this thread</a> to try to figure out the issue, but so far there's no result.</p>

<p>My setup:</p>

<p><code>$HADOOP_HOME</code> is set to <code>/usr/local/cellar/hadoop/2.7.1</code> where my hadoop files are</p>

<p><code>$SPARK_HOME</code> is set to <code>/usr/local/cellar/apache_spark/1.5.2</code></p>

<p>I also followed those threads and guide online as close as possible to get </p>

<pre><code>export PYTHONPATH=$SPARK_HOME/libexec/python:$SPARK_HOME/libexec/python/build:$PYTHONPATH

export PATH=$PATH:$HADOOP_HOME/bin

PYTHONPATH=$SPARK_HOME/libexec/python/lib/py4j-0.8.2.1-src.zip:$PYTHONPATH
</code></pre>

<p>then I used this piece of code to test in the first thread I linked</p>

<pre><code>from pyspark import SparkContext, SparkConf
import pymongo_spark

pymongo_spark.activate()

def main():
    conf = SparkConf().setAppName('pyspark test')
    sc = SparkContext(conf=conf)
if __name__ == '__main__':
    main()
</code></pre>

<p>Then in the terminal, I did:</p>

<pre><code>$SPARK_HOME/bin/spark-submit --jars $HADOOP_HOME/libexec/share/hadoop/mapreduce/mongo-hadoop-r1.4.2-1.4.2.jar --driver-class-path $HADOOP_HOME/libexec/share/hadoop/mapreduce/mongo-hadoop-r1.4.2-1.4.2.jar --master local[4] ~/Documents/pysparktest.py
</code></pre>

<p>Where <code>mongo-hadoop-r1.4.2-1.4.2.jar</code> is the jar I built following <a href="https://github.com/mongodb/mongo-hadoop#building" rel="nofollow noreferrer">this guide</a></p>

<p>I'm definitely missing things, but I'm not sure where/what I'm missing. I'm running everything locally on Mac OSX El Capitan. Almost sure this doesn't matter, but just wanna add it in.</p>

<p>EDIT:</p>

<p>I also used another jar file <code>mongo-hadoop-1.5.0-SNAPSHOT.jar</code>, the same problem remains</p>

<p>my command:</p>

<pre><code>$SPARK_HOME/bin/spark-submit --jars $HADOOP_HOME/libexec/share/hadoop/mapreduce/mongo-hadoop-1.5.0-SNAPSHOT.jar --driver-class-path $HADOOP_HOME/libexec/share/hadoop/mapreduce/mongo-hadoop-1.5.0-SNAPSHOT.jar --master local[4] ~/Documents/pysparktest.py
</code></pre>