PySpark: Calculate grouped-by AUC
<ul>
<li>Spark version: 1.6.0</li>
</ul>

<p>I tried computing AUC (area under ROC) grouped by the field <code>id</code>. Given the following data:</p>

<pre><code># Within each key-value pair
# key is "id"
# value is a list of (score, label)
data = sc.parallelize(
         [('id1', [(0.5, 1.0), (0.6, 0.0), (0.7, 1.0), (0.8, 0.0)),
          ('id2', [(0.5, 1.0), (0.6, 0.0), (0.7, 1.0), (0.8, 0.0))
         ]
</code></pre>

<p>The <a href="http://spark.apache.org/docs/latest/api/python/pyspark.mllib.html#pyspark.mllib.evaluation.BinaryClassificationMetrics" rel="nofollow">BinaryClassificationMetrics</a> class can calculate the AUC given a list of <code>(score, label)</code>.</p>

<p>I want to compute AUC by key (i.e. <code>id1, id2</code>). But how to "map" a <code>class</code> to an RDD by key?</p>

<h3>Update</h3>

<p>I tried to wrap the <code>BinaryClassificationMetrics</code> in a function:</p>

<pre><code>def auc(scoreAndLabels):
    return BinaryClassificationMetrics(scoreAndLabels).areaUnderROC
</code></pre>

<p>And then map the wrapper function to each values:</p>

<pre><code>data.groupByKey()\
    .mapValues(auc)
</code></pre>

<p>But the list of <code>(score, label)</code> is in fact of type <code>ResultIterable</code> in <code>mapValues()</code> while the <code>BinaryClassificationMetrics</code> expects <code>RDD</code>.</p>

<p>Is there any approach of converting the <code>ResultIterable</code> to <code>RDD</code> so the the <code>auc</code> function can be applied? Or any other workaround for computing group-by AUC (without importing third-party modules like scikit-learn)?</p>