Select specific columns in a PySpark dataframe to improve performance
<p>Working with Spark dataframes imported from Hive, sometimes I end up with several columns that I don't need. Supposing that I don't want to filter them with </p>

<pre><code>df = SqlContext.sql('select cols from mytable')
</code></pre>

<p>and I'm importing the entire table with </p>

<pre><code>df = SqlContext.table(mytable)
</code></pre>

<p>does a <code>select</code> and subsequent <code>cache</code> improves performance/decrease memory usage, like</p>

<pre><code>df = df.select('col_1', 'col_2', 'col_3')
df.cache()
df.count()
</code></pre>

<p>or is just waste of time? I will do lots of operations and data manipulations on <code>df</code>, like <code>avg</code>, <code>withColumn</code>, etc.</p>