How to get the condensed form of pairwise distances directly?
<p>I have a very large scipy sparse csr matrix.  It is a 100,000x2,000,000 dimensional matrix. Let's call it <code>X</code>. Each row is a sample vector in a 2,000,000 dimensional space.</p>

<p>I need to calculate the cosine distances between each pair of samples very efficiently. I have been using <code>sklearn pairwise_distances</code> function with a subset of vectors in <code>X</code> which gives me a dense matrix D: the square form of the pairwise distances which contains redundant entries.  How can I use <code>sklearn pairwise_distances</code> to get the condensed form directly?  Please refer to <a href="http://docs.scipy.org/doc/scipy/reference/generated/scipy.spatial.distance.pdist.html" rel="nofollow">http://docs.scipy.org/doc/scipy/reference/generated/scipy.spatial.distance.pdist.html</a> to see what the condensed form is.  It is the output of <code>scipy pdist</code> function.</p>

<p>I have memory limitations and I can't calculate the square form and then get the condensed form.  Due to memory limitations, I also cannot use <code>scipy pdist</code> as it requires a dense matrix <code>X</code> which does not again fit in memory. I thought about looping through different chunks of <code>X</code> and calculate the condensed form for each chunk and join them together to get the complete condensed form, but this is relatively cumbersome.  Any better ideas?</p>

<p>Any help is much much appreciated. Thanks in advance.</p>

<p>Below is a reproducible example (of course for demonstration purposes <code>X</code> is much smaller):</p>

<pre><code>from scipy.sparse import rand
from scipy.spatial.distance import pdist
from sklearn.metrics.pairwise import pairwise_distances
X = rand(1000, 10000, density=0.01, format='csr')
dist1 = pairwise_distances(X, metric='cosine')
dist2 = pdist(X.A, 'cosine')
</code></pre>

<p>As you see <code>dist2</code> is in the condensed form and is a 499500 dimensional vector. But <code>dist1</code> is in the symmetric squareform and is a 1000x1000 matrix.</p>