How do Dask dataframes handle larger-than-memory datasets?
<p>The <a href="http://dask.pydata.org/en/latest/dataframe.html" rel="noreferrer">documentation of the Dask package for dataframes</a> says:</p>

<blockquote>
  <p>Dask dataframes look and feel like pandas dataframes, but operate on
  datasets larger than memory using multiple threads.</p>
</blockquote>

<p>But later in the same page:</p>

<blockquote>
  <p>One dask DataFrame is comprised of several in-memory pandas DataFrames
  separated along the index.</p>
</blockquote>

<p>Does Dask read the different DataFrame partitions from disk sequentally and perform computations to fit into memory? Does it spill some partitions to disk when needed? In general, how does Dask manage the memory &lt;--> disk IO of data to allow larger-than-memory data analysis?</p>

<p>I tried to perform some basic computations (e.g. mean rating) on the 10M MovieLens dataset and my laptop (8GB RAM) started to swap.</p>