How to partition Spark RDD when importing Postgres using JDBC?
<p>I am importing a Postgres database into Spark. I know that I can partition on import, but that requires that I have a numeric column (I don't want to use the <code>value</code> column because it's all over the place and doesn't maintain order):</p>

<pre><code>df = spark.read.format('jdbc').options(url=url, dbtable='tableName', properties=properties).load()
df.printSchema()

root
 |-- id: string (nullable = false)
 |-- timestamp: timestamp (nullable = false)
 |-- key: string (nullable = false)
 |-- value: double (nullable = false)
</code></pre>

<p>Instead, I am converting the dataframe into an rdd (of enumerated tuples) and trying to partition that instead:</p>

<pre><code>rdd = df.rdd.flatMap(lambda x: enumerate(x)).partitionBy(20)
</code></pre>

<p>Note that I used <code>20</code> because I have 5 workers with one core each in my cluster, and <code>5*4=20</code>.</p>

<p>Unfortunately, the following command still takes forever to execute:</p>

<pre><code>result = rdd.first()
</code></pre>

<p>Therefore I am wondering if my logic above makes sense? Am I doing anything wrong? From the web GUI, it looks like the workers are not being used:</p>

<p><a href="https://i.stack.imgur.com/Mvd9Z.png" rel="noreferrer"><img src="https://i.stack.imgur.com/Mvd9Z.png" alt="enter image description here"></a></p>