Memory and time efficient competition of correlation between each column in a large matrix and a vector
<p>This question extends <a href="https://stackoverflow.com/questions/40141738/is-there-a-fast-estimation-of-simple-regression-a-regression-line-with-only-int">this post</a>, relating to a <code>machine learning</code> <code>feature selection</code> procedure where I have a large matrix of features and I'd like to perform a fast and crude <code>feature selection</code> by measuring the <code>correlation</code> between the outer product between each pair of features and the response, since I'll be using a <code>random forest</code> or <code>boosting</code> <code>classifier</code>.</p>

<p>The number of features is ~60,000 and the number of responses is ~2,200,000.</p>

<p>Given unlimited memory perhaps the fastest way to go about this would be to generate a <code>matrix</code> where the columns are the outer products of all pairs of features and use <code>cor</code> of that <code>matrix</code> against the response. As a smaller dimension example:</p>

<pre><code>set.seed(1)
feature.mat &lt;- matrix(rnorm(2200*100),nrow=2200,ncol=100)
response.vec &lt;- rnorm(2200)

#generate indices of all unique pairs of features and get the outer products:
feature.pairs &lt;- t(combn(1:ncol(feature.mat),2))
feature.pairs.prod &lt;- feature.mat[,feature.pairs[,1]]*feature.mat[,feature.pairs[,2]]

#compute the correlation coefficients
res &lt;- cor(feature.pairs.prod,response.vec)
</code></pre>

<p>But for my real dimensions <code>feature.pairs.prod</code> is 2,200,000 by 1,799,970,000 which obviously cannot be stored in memory.</p>

<p>So my question is if and how it is possible to get all the correlations in reasonable computation time?</p>

<p>I was thinking that perhaps breaking down <code>feature.pairs.prod</code> to chunks that fit in memory, and then do <code>cor</code> between them and <code>response.vec</code> one at a time will be the fastest but I'm not sure how to automatically test in <code>R</code> what dimensions I need these chunks to be.</p>

<p>Another option is to <code>apply</code> a function over <code>feature.pairs</code> which will compute the outer product and then <code>cor</code> between that and <code>response.vec</code>.</p>

<p>Any suggestions?</p>