How to use dask to quickly access subsets of the data?
<p>One of the main reasons I love pandas is that it's easy to home in on subsets, e.g. <code>df[df.sample.isin(['a', 'c', 'p'])]</code> or <code>df[df.age &lt; 35]</code>. Is dask dataframe good at (optimized for) this as well? The tutorials I've seen have focused on whole-column manipulations. </p>

<p>My specific application is (thousands of named GCMS samples) x (~20000 time points per sample) x (500 m/z channels) x (intensity), and I'm looking for the fastest tool to pull arbitrary subsets, e.g. </p>

<pre><code>df[df.sample.isin([...]) &amp; df.rt.lt(800) &amp; df.rt.gt(600) &amp; df.mz.isin(...)]
</code></pre>

<p>If dask is a good choice, then I would appreciate advice on how best to structure it. </p>

<h2>What I've tried</h2>

<p>What I've tried so far is to convert each sample to pandas dataframe that looks like</p>

<pre><code>                  smp     rt       14       15       16       17       18  
0  160602_JK_OFCmix:1  271.0  64088.0   9976.0  26848.0  23928.0  89600.0   
1  160602_JK_OFCmix:1  271.1  65472.0  10880.0  28328.0  24808.0  91840.0   
2  160602_JK_OFCmix:1  271.2  64528.0  10232.0  27672.0  25464.0  90624.0   
3  160602_JK_OFCmix:1  271.3  63424.0  10272.0  27600.0  25064.0  90176.0   
4  160602_JK_OFCmix:1  271.4  64816.0  10640.0  27592.0  24896.0  90624.0  
</code></pre>

<p>('smp' is sample name, 'rt' is retention time, 14,15,...500 are m/z channels), save to hdf with zlib, level=1, then make the dask dataframe with </p>

<pre><code>ddf = dd.read_hdf(*.hdf5, key='/*', chunksize=100000, lock=False)
</code></pre>

<p>but <code>df = ddf[ddf.smp.isin([...a couple of samples...]).compute()</code> is 100x slower than <code>ddf['57'].mean().compute()</code>. </p>

<p>(Note: this is with <code>dask.set_options(get=dask.multiprocessing.get)</code>)</p>