Spark code takes too long to run
<p>I have a page of code that takes hours to run and I was hoping for no more than a few minutes. I watched the console while it ran and it appears to be especially hung up on this section</p>

<pre><code>def getListComponents(p):
    pair = []
    for i in range(len(p)):
        pair.append((i, p[i]))
    return pair

def addPoints(p1, p2):
    summedPoint = []
    for i in range(len(p1)):
        summedPoint.append(p1[i] + p2[i])
    return summedPoint

for i in range(len(newCentroids)): # len(newCentroids) = 4
    (cnumerator
        .append(wTmx.map(lambda p: getListComponents(p))
        .map(lambda  p: p[i])
        .reduceByKey(lambda p1, p2 : addPoints(p1, p2))))
</code></pre>

<p>I have read that increasing the memory might help and I've tried anywhere from 2gs to 4gs, but saw no improvement.</p>

<pre><code>--executor-memory=3g
</code></pre>

<p>I have also read that changing the default parallelism can help, setting 2-3 per machine being ideal (I am running a cluster of three)</p>

<pre><code>--default-parallelism=9
</code></pre>

<p>but this has had no effect.</p>

<p>So now I am assuming the problem is in my apparent inability to write efficient code. Is there a way for me to change this code to do the same thing (the entire program works as expected), but take much less time?</p>

<p>Thanks.</p>