Scipy: sparse matrix conditional removal of columns
<p>I have a large (79 000 x 480 000) sparse csr matrix. I am trying to remove all columns (within a certain range) for which each value &lt; k.</p>

<p>In regular numpy matrices this is simply done by a mask:</p>

<pre><code>m = np.array([[0,2,1,1],
                [0,4,2,0],
                [0,3,4,0]])
mask = (arr &lt; 2)
idx = mask.all(axis=0)
result = m[:, ~idx]
print result
&gt;&gt;&gt; [[2 1]
     [4 2]
     [3 4]]
</code></pre>

<p>The unary bitwise negation operator ~ and boolean mask functionality are not available for sparse matrices however. What is the best method to:</p>

<ol>
<li>Obtain the indices of columns where all values fulfill condition e &lt; k.</li>
<li>Remove these columns based on the list of indices.</li>
</ol>

<p>Some things to note:</p>

<ol>
<li>The columns represent ngram text features: there are no columns in the matrix for which each element is zero. </li>
</ol>

<p>Is using the csr matrix format even a plausible choice for this?
Do I transpose and make use of .nonzero()? I have a fair amount of working memory (192GB) so time efficiency is preferable to memory efficiency.</p>