Spark: Computing correlations of a DataFrame with missing values
<p>I currently have a DataFrame of doubles with approximately 20% of the data being null values. I want to calculate the Pearson correlation of one column with every other column and return the columnId's of the top 10 columns in the DataFrame. </p>

<p>I want to filter out nulls using pairwise deletion, similar to R's pairwise.complete.obs option in its Pearson correlation function. That is, if one of the two vectors in any correlation calculation has a null at an index, I want to remove that row from both vectors.</p>

<p>I currently do the following:</p>

<pre><code>val df = ... //my DataFrame
val cols = df.columns
df.registerTempTable("dataset")
val target = "Row1"
val mapped = cols.map {colId =&gt; 
    val results = sqlContext.sql(s"SELECT ${target}, ${colId} FROM dataset WHERE (${colId} IS NOT NULL AND ${target} IS NOT NULL)")
    (results.stat.corr(colId, target) , colId)
  }.sortWith(_._1 &gt; _._1).take(11).map(_._2)
</code></pre>

<p>This runs very slowly, as every single map iteration is its own job. Is there a way to do this efficiently, perhaps using Statistics.corr in the Mllib, as per <a href="https://stackoverflow.com/questions/35849435/spark-1-6-pearson-correlation">this SO Question (Spark 1.6 Pearson Correlation)</a></p>