Spark coalesce vs collect, which one is faster?
<p>I am using <code>pyspark</code> to process 50Gb data using AWS EMR with ~15 m4.large cores. </p>

<p>Each row of the data contains some information at a specific time on a day. I am using the following <code>for</code> loop to extract and aggregate information for every hour. Finally I <code>union</code> the data, as I want my result to save in <strong>one</strong> csv file.</p>

<pre><code># daily_df is a empty pyspark DataFrame
for hour in range(24):
    hourly_df = df.filter(hourFilter("Time")).groupby("Animal").agg(mean("weights"), sum("is_male"))
    daily_df = daily_df.union(hourly_df)
</code></pre>

<p>As of my knowledge, I have to perform the following to force the <code>pyspark.sql.Dataframe</code> object to save to 1 csv files (approx 1Mb) instead of 100+ files:</p>

<pre><code>daily_df.coalesce(1).write.csv("some_local.csv")
</code></pre>

<p>It seems it took about 70min to finish this progress, and I am wondering if I can make it faster by using <code>collect()</code> method like?</p>

<pre><code>daily_df_pandas = daily_df.collect()
daily_df_pandas.to_csv("some_local.csv")
</code></pre>