At what situation I can use Dask instead of Apache Spark?
<p>I am currently using Pandas and Spark for data analysis. I found Dask provides parallelized NumPy array and Pandas DataFrame.</p>

<p>Pandas is easy and intuitive for doing data analysis in Python. But I find difficulty in handling multiple bigger dataframes in Pandas due to limited system memory.</p>

<blockquote>
  <p>Simple Answer:</p>
  
  <p>Apache Spark is an all-inclusive framework combining distributed
  computing, SQL queries, machine learning, and more that runs on the
  JVM and is commonly co-deployed with other Big Data frameworks like
  Hadoop. ... Generally Dask is smaller and lighter weight than Spark.</p>
</blockquote>

<p>I get to know below details from <a href="http://dask.pydata.org/en/latest/spark.html" rel="noreferrer">http://dask.pydata.org/en/latest/spark.html</a></p>

<ul>
<li>Dask is light weighted</li>
<li>Dask is typically used on a single machine, but also runs well on a distributed cluster.</li>
<li>Dask to provides parallel arrays, dataframes, machine learning, and custom algorithms</li>
<li>Dask has an advantage for Python users because it is itself a Python library, so serialization and debugging when things go wrong happens more smoothly.</li>
<li>Dask gives up high-level understanding to allow users to express more complex parallel algorithms. </li>
<li>Dask is lighter weight and is easier to integrate into existing code and hardware.</li>
<li>If you want a single project that does everything and you’re already on Big Data hardware then Spark is a safe bet</li>
<li>Spark is typically used on small to medium sized cluster but also runs well on a single machine.</li>
</ul>

<p>I understand more things about Dask from the below link
<a href="https://www.continuum.io/blog/developer-blog/high-performance-hadoop-anaconda-and-dask-your-cluster" rel="noreferrer">https://www.continuum.io/blog/developer-blog/high-performance-hadoop-anaconda-and-dask-your-cluster</a></p>

<ul>
<li>If you’re running into memory issues, storage limitations, or CPU boundaries on a single machine when using Pandas, NumPy, or other computations with Python, Dask can help you scale up on all of the cores on a single machine, or scale out on all of the cores and memory across your cluster.</li>
<li>Dask works well on a single machine to make use of all of the cores on your laptop and process larger-than-memory data</li>
<li>scales up resiliently and elastically on clusters with hundreds of nodes.</li>
<li>Dask works natively from Python with data in different formats and storage systems, including the Hadoop Distributed File System (HDFS) and Amazon S3. Anaconda and Dask can work with your existing enterprise Hadoop distribution, including Cloudera CDH and Hortonworks HDP.</li>
</ul>

<p><a href="http://dask.pydata.org/en/latest/dataframe-overview.html" rel="noreferrer">http://dask.pydata.org/en/latest/dataframe-overview.html</a></p>

<p><strong>Limitations</strong></p>

<p>Dask.DataFrame does not implement the entire Pandas interface. Users expecting this will be disappointed.Notably, dask.dataframe has the following limitations:</p>

<ol>
<li>Setting a new index from an unsorted column is expensive</li>
<li>Many operations, like groupby-apply and join on unsorted columns require setting the index, which as mentioned above, is expensive</li>
<li>The Pandas API is very large. Dask.dataframe does not attempt to implement many pandas features or any of the more exotic data structures like NDFrames</li>
</ol>

<p>Thanks to the Dask developers. It seems like very promising technology.</p>

<p>Overall I can understand Dask is simpler to use than spark. Dask is as flexible as Pandas with more power to compute with more cpu's parallely.</p>

<p>I understand all the above facts about Dask.</p>

<p><strong>So, roughly how much amount of data(in terabyte) can be processed with Dask?</strong></p>