How to setup Apache Spark to use local hard disk when data does not fit in RAM in local mode?
<p>I have 50 GB dataset which doesn't fit in 8 GB RAM of my work computer but it has 1 TB local hard disk. </p>

<p>The below link from offical documentation mentions that Spark can use local hard disk if data doesnt fit in the memory.</p>

<p><a href="http://spark.apache.org/docs/latest/hardware-provisioning.html" rel="nofollow">http://spark.apache.org/docs/latest/hardware-provisioning.html</a></p>

<blockquote>
  <p><strong>Local Disks</strong></p>
  
  <p>While Spark can perform a lot of its computation in memory, it still
  uses local disks to store data that doesn’t fit in RAM, as well as to
  preserve intermediate output between stages.</p>
</blockquote>

<p>For me computational time is not at all a priority but fitting the data into a single computer's RAM/hard disk for processing is more important due to lack of alternate options.</p>

<p>Note:
I am looking for a solution which <strong><em>doesn't</em></strong> include the below items</p>

<ol>
<li>Increase the RAM</li>
<li>Sample &amp; reduce data size</li>
<li>Use cloud or cluster computers</li>
</ol>

<p>My end objective is to use Spark MLLIB to build machine learning models.
I am looking for real-life, practical solutions that people successfully used Spark to operate on data that doesn't fit in RAM in standalone/local mode in a single computer. Have someone done this successfully without major limitations?</p>

<p><strong>Questions</strong></p>

<ol>
<li><p><a href="http://www.sas.com" rel="nofollow">SAS</a> have similar capability of out-of-core processing using which it can use both RAM &amp; local hard disk for model building etc. Can Spark be made to work in the same way when data is more than RAM size?</p></li>
<li><p>SAS writes persistent the complete dataset to hardisk in ".sas7bdat" format can Spark do similar persistent to hard disk?</p></li>
<li>If this is possible, how to install and configure Spark for this purpose?</li>
</ol>