Spark on localhost
<p>For testing purposes, while I don´t have production cluster, I am using spark locally:</p>

<pre><code>print('Setting SparkContext...')
sconf = SparkConf()
sconf.setAppName('myLocalApp')
sconf.setMaster('local[*]')
sc = SparkContext(conf=sconf)
print('Setting SparkContext...OK!')
</code></pre>

<p>Also, I am using a very very small dataset, consisting of only 20 rows in a postgresql database ( ~2kb)</p>

<p>Also(!), my code is quite simple as well, only grouping <strong>20 rows</strong> by a key and applying a trivial map operation</p>

<pre><code>params = [object1, object2]
rdd = df.rdd.keyBy(lambda x: (x.a, x.b, x.c)) \
                          .groupByKey() \
                          .mapValues(lambda value: self.__data_interpolation(value, params))


def __data_interpolation(self, data, params):
    # TODO: only for testing
    return data
</code></pre>

<p>What bothers me is that the whole execution takes about <strong>5 minutes!!</strong></p>

<p>Inspecting the Spark UI, I see that most of the time was spent in Stage 6: byKey method. (Stage 7, collect() method was also slow...)</p>

<p>Some info:</p>

<p><a href="https://i.stack.imgur.com/0j6aC.png" rel="noreferrer"><img src="https://i.stack.imgur.com/0j6aC.png" alt="enter image description here"></a></p>

<p><a href="https://i.stack.imgur.com/YK7Ul.png" rel="noreferrer"><img src="https://i.stack.imgur.com/YK7Ul.png" alt="enter image description here"></a></p>

<p>These numbers make no sense to me... Why do I need 22 tasks, executing for 54 sec, to process less than 1 kb of data </p>

<p>Can it be a network issue, trying to figure out the ip address of localhost?
I don't know... Any clues?</p>