spark.default.parallelism for Parallelize RDD defaults to 2 for spark submit
<p>Spark standalone cluster with a master and 2 worker nodes 4 cpu core on each worker. Total 8 cores for all workers.</p>

<p>When running the following via spark-submit (spark.default.parallelism is not set)</p>

<pre class="lang-scala prettyprint-override"><code>val myRDD = sc.parallelize(1 to 100000)
println("Partititon size - " + myRDD.partitions.size)
val totl = myRDD.reduce((x, y) =&gt; x + y)
println("Sum - " + totl)
</code></pre>

<p>It returns value 2 for partition size.</p>

<p>When using spark-shell  by connecting to spark standalone cluster the same code returns correct partition size 8.  </p>

<p>What can be the reason ?</p>

<p>Thanks. </p>