All of the large-scale Dask collections like Dask Array, Dask DataFrame, and Dask Bag and the fine-grained APIs like delayed and futures generate task graphs where each node in the graph is a normal Python function and edges between nodes are normal Python objects that are created by one task as outputs and used as inputs in another task. After Dask generates these task graphs, it needs to execute them on parallel hardware. This is the job of a task scheduler. Different task schedulers exist, and each will consume a task graph and compute the same result, but with different performance characteristics.

Dask collections and schedulers

Dask has two families of task schedulers:



Local Threads

import dask

The threaded scheduler executes computations with a local multiprocessing.pool.ThreadPool. It is lightweight and requires no setup. It introduces very little task overhead (around 50us per task) and, because everything occurs in the same process, it incurs no costs to transfer data between tasks. However, due to Python’s Global Interpreter Lock (GIL), this scheduler only provides parallelism when your computation is dominated by non-Python code, as is primarily the case when operating on numeric data in NumPy arrays, Pandas DataFrames, or using any of the other C/C++/Cython based projects in the ecosystem.

The threaded scheduler is the default choice for Dask Array, Dask DataFrame, and Dask Delayed. However, if your computation is dominated by processing pure Python objects like strings, dicts, or lists, then you may want to try one of the process-based schedulers below (we currently recommend the distributed scheduler on a local machine).

Local Processes



import dask.multiprocessing



This is common in basic data ingestion workloads, such as those are common in Dask Bag, where the multiprocessing scheduler is the default:

>>> import dask.bag as db


Single Thread

import dask


For example, when using IPython or Jupyter notebooks, the %debug, %pdb, or %prun magics will not work well when using the parallel Dask schedulers (they were not designed to be used in a parallel computing context). However, if you run into an exception and want to step into the debugger, you may wish to rerun your computation under the single-threaded scheduler where these tools will function properly.

Dask Distributed (local)

from dask.distributed import Client
client = Client()
# or
client = Client(processes=False)

The Dask distributed scheduler can either be setup on a cluster or run locally on a personal machine. Despite having the name “distributed”, it is often pragmatic on local machines for a few reasons:

  1. It provides access to asynchronous API, notably Futures

You can read more about using the Dask distributed scheduler on a single machine in these docs.

Dask Distributed (Cluster)

You can also run Dask on a distributed cluster. There are a variety of ways to set this up depending on your cluster. We recommend referring to the setup documentation for more information.


You can configure the global default scheduler by using the dask.config.set(scheduler...) command. This can be done globally:



or as a context manager:

with dask.config.set(scheduler='threads'):

or within a single compute call:


Additionally some of the scheduler support other keyword arguments. For example, the pool-based single-machine scheduler allows you to provide custom pools or specify the desired number of workers:

from multiprocessing.pool import ThreadPool
with dask.config.set(pool=ThreadPool(4)):

with dask.config.set(num_workers=4):