read_hdf where fails 'all of the variables refrences must be a reference to an axis...'
<p>Stuck on the following.   </p>

<pre><code>log_iter = pd.read_hdf(FN, dspath, 
                       where = [pd.Term('hashID','=',idList)],
                       iterator=True, 
                       chunksize=3000)
</code></pre>

<p>The dspath has 35 columns and can be quite large causing MemoryError.   </p>

<p>So trying to go the iteator/chunksize route.   But the 'where=' clause is failing with </p>

<pre><code>ValueError: The passed where expression: [hashID=[147685,...,147197]]
        contains an invalid variable reference
        all of the variable refrences must be a reference to
        an axis (e.g. 'index' or 'columns'), or a data_column
        The currently defined references are: ** list of column names **
</code></pre>

<p>The problem is that hashID is not in the list of column names.  Yet, if I do </p>

<pre><code>read_hdf(FN, dspath).columns
</code></pre>

<p>The hashID is in the columns. Any suggestions?  My goal is to read in all rows x 35 columns whose hashID is in idList. </p>

<p>Update.  The following works and shows that the hashID exists as a column once the dataset is read in. </p>

<pre><code>def dsIterator(self, q, idList):
    hID = u'hashID'
    FN = self.db._hdf_FN()
    dspath = self.getdatasetname(q)
    log_iter = pd.read_hdf(FN, dspath, 
                           #where = [pd.Term(u'logid_hashID','=',idList)],
                           iterator=True, 
                           chunksize=30000)
    n_all = 0
    retDF = None
    for dfChunk in log_iter:
        goodChunk = dfChunk.loc[dfChunk[hID].isin(idList)]
        if retDF is None : retDF = goodChunk
        else: 
            retDF = pd.concat([retDF, goodChunk], ignore_index=True)
        n_all += dfChunk[hID].count()
    n_ret = retDF[hID].count()
    return retDF
</code></pre>