pyspark using one task for mapPartitions when converting rdd to dataframe
<p>I'm confused as to why it appears that Spark is using 1 task for <code>rdd.mapPartitions</code> when converting the resulting RDD to a DataFrame. </p>

<p>This is an issue for me because I would like to go from :</p>

<p><code>DataFrame</code> --> <code>RDD</code> --> <code>rdd.mapPartitions</code> --> <code>DataFrame</code> </p>

<p>so that I can read in data (DataFrame), apply a non-SQL function to chunks of data (mapPartitions on RDD) and then convert back to a DataFrame so that I can using the <code>DataFrame.write</code> process. </p>

<p>I am able to go from DataFrame --> mapPartitions and then use an RDD writer like saveAsTextFile but that is less than ideal since the <code>DataFrame.write</code> process can do things like overwrite and save data in Orc format. So I'd like to learn <em>why</em> this is going on, but from a pratical perspective I'm primarily concerned with being able to just go from a DataFrame --> mapParitions --> to using the DataFrame.write process.</p>

<p>Here is a reproducible example. The following works as expected, with 100 tasks for the <code>mapPartitions</code> work:</p>

<pre><code>from pyspark.sql import SparkSession
import pandas as pd

spark = SparkSession \
    .builder \
    .master("yarn-client") \
    .enableHiveSupport() \
    .getOrCreate()

sc = spark.sparkContext

df = pd.DataFrame({'var1':range(100000),'var2': [x-1000 for x in range(100000)]})
spark_df = spark.createDataFrame(df).repartition(100)

def f(part):
    return [(1,2)]

spark_df.rdd.mapPartitions(f).collect()
</code></pre>

<p>However if the last line is change to something like <code>spark_df.rdd.mapPartitions(f).toDF().show()</code> then there will only be one task for the <code>mapPartitions</code> work.</p>

<p>Some screenshots illustrating this below:
<a href="https://i.stack.imgur.com/g2sn0.png" rel="noreferrer"><img src="https://i.stack.imgur.com/g2sn0.png" alt="enter image description here"></a>
<a href="https://i.stack.imgur.com/CoeNX.png" rel="noreferrer"><img src="https://i.stack.imgur.com/CoeNX.png" alt="enter image description here"></a></p>