Spark SQL: Cache Memory footprint improves with 'order by'
<p>I have two scenarios where I have <code>23 GB</code> partitioned <code>parquet</code> data and reading few of the <code>columns</code> &amp; <code>caching</code> it upfront to fire a series of subsequent queries later on. </p>

<p><strong>Setup</strong>:</p>

<ul>
<li>Cluster: 12 Node EMR </li>
<li>Spark Version: 1.6 </li>
<li>Spark Configurations: Default</li>
<li>Run Configurations: Same for both cases</li>
</ul>

<p><strong>Case 1</strong>:</p>

<pre><code>val paths = Array("s3://my/parquet/path", ...)
val parqFile = sqlContext.read.parquet(paths:_*)
parqFile.registerTempTable("productViewBase")
val dfMain = sqlContext.sql("select guid,email,eventKey,timestamp,pogId from productViewBase")
dfMain.cache.count
</code></pre>

<p>From <code>SparkUI</code>, the input data read is 6.2 GB and the cached object is of <strong>15.1 GB</strong>.</p>

<p><strong>Case 1</strong>:</p>

<pre><code>val paths = Array("s3://my/parquet/path", ...)
val parqFile = sqlContext.read.parquet(paths:_*)
parqFile.registerTempTable("productViewBase")
val dfMain = sqlContext.sql("select guid,email,eventKey,timestamp,pogId from productViewBase order by pogId")
dfMain.cache.count
</code></pre>

<p>From <code>SparkUI</code>, the input data read is 6.2 GB and the cached object is of <strong>5.5 GB</strong>.</p>

<p>Any explanation, or code-reference to this behavior? </p>