How to connect spark with cassandra using spark-cassandra-connector?
<p>You must forgive my noobness but I'm trying to setup a spark cluster that connects to cassandra running a python script, currently I am using datastax enterprise to run cassandra on solr search mode. I understand that, in order to use the spark-cassandra connector that datastax provides, you must run cassandra in analytics mode (using -k option). Currently I have got it to work only using the dse spark version, for which, to make it work I followed the next steps:</p>

<ol>
<li>Start dse cassandra in analytics mode </li>
<li>change $PYTHONPATH env variable to /path/to/spark/dse/python:/path/to/spark/dse/python/lib/py4j-*.zip:$PYTHONPATH </li>
<li>run as root the standalone script with <code>python test-script.py</code></li>
</ol>

<p>Besides, I made another test using the spark alone (not dse version), trying to include the java packages that make driver classes accesible, I did: </p>

<ol>
<li>Add spark.driver.extraClassPath = /path/to/spark-cassandra-connector-SNAPSHOT.jar to the file spark-defaults.conf
2.execute <code>$SPARK_HOME/bin/spark-submit  —packages com.datastax.spark:spark-cassandra...</code> </li>
</ol>

<p>I also tried running pyspark shell and test if sc had the method cassandraTable to see if the driver was loaded but didn't work out, in both cases I get the following error message:</p>

<pre><code>AttributeError: 'SparkContext' object has no attribute 'cassandraTable'
</code></pre>

<p>My goal is to undestand what I must do to make the non-dse spark version connect with cassandra and have the methods from the driver available.</p>

<p>I also want to know if it is possible to use the dse spark-cassandra connector with a cassandra node that is NOT running with dse. </p>

<p>Thanks for your help</p>