Running a Spark job with spark-submit across the whole cluster
<p>I have recently set up a Spark cluster on Amazon EMR with 1 master and 2 slaves.</p>

<p>I can run <code>pyspark</code>, and submit jobs with <code>spark-submit</code>.</p>

<p>However, when I create a standalone job, like <code>job.py</code>, I create a SparkContext, like so:</p>

<pre><code>sc=SparkContext("local", "App Name")
</code></pre>

<p>This doesn't seem right, but I'm not sure what to put there.</p>

<p>When I submit the job, I am sure it is not utilizing the whole cluster.</p>

<p>If I want to run a job against my entire cluster, say 4 processes per slave, what do I have to</p>

<p>a.) pass as arguments to <code>spark-submit</code></p>

<p>b.) pass as arguments to <code>SparkContext()</code> in the script itself.</p>