Computing precision and recall for two sets of keywords in NLTK and Scikit for sets of different sizes
<p>I am trying to compute precision and recall for two sets of keywords. The <code>gold_standard</code> has 823 terms and the <code>test</code> has 1497 terms.</p>

<p>Using <code>nltk.metrics</code>'s version of <code>precision</code> and <code>recall</code>, I am able to provide the two sets just fine. But doing the same for Scikit is throwing me an error:</p>

<blockquote>
  <p>ValueError: Found arrays with inconsistent numbers of samples: [ 823 1497]</p>
</blockquote>

<p>How do I resolve this?</p>

<pre><code>#!/usr/bin/python3

from nltk.metrics import precision, recall
from sklearn.metrics import precision_score
from sys import argv
from time import time
import numpy
import csv

def readCSVFile(filename):
    termList = set()
    with open(filename, 'rt', encoding='utf-8') as f:
        reader = csv.reader(f)
        for row in reader:
            termList.update(row)
    return termList

def readDocuments(gs_file, fileToProcess):
    print("Reading CSV files...")
    gold_standard = readCSVFile(gs_file)
    test = readCSVFile(fileToProcess)
    print("All files successfully read!")
    return gold_standard, test

def calcPrecisionScipy(gs, test):
    gs = numpy.array(list(gs))
    test = numpy.array(list(test))
    print("Precision Scipy: ",precision_score(gs, test, average=None))

def process(datasest):
    print("Processing input...")
    gs, test = dataset
    print("Precision: ", precision(gs, test))
    calcPrecisionScipy(gs, test)

def usage():
    print("Usage: python3 generate_stats.py gold_standard.csv termlist_to_process.csv")

if __name__ == '__main__':
    if len(argv) != 3:
        usage()
        exit(-1)

    t0 = time()
    process(readDocuments(argv[1], argv[2]))
    print("Total runtime: %0.3fs" % (time() - t0))
</code></pre>

<p>I referred to the following pages for coding:</p>

<ul>
<li><a href="http://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_recall_fscore_support.html" rel="nofollow">http://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_recall_fscore_support.html</a></li>
<li><a href="http://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_score.html#sklearn.metrics.precision_score" rel="nofollow">http://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_score.html#sklearn.metrics.precision_score</a></li>
</ul>

<p>=================================Update===================================</p>

<p>Okay, so I tried to add 'non-sensical' data to the list to make them equal length:</p>

<pre><code>def calcPrecisionScipy(gs, test):
    if len(gs) &lt; len(test):
        gs.update(list(range(len(test)-len(gs))))
    gs = numpy.array(list(gs))
    test = numpy.array(list(test))
    print("Precision Scipy: ",precision_score(gs, test, average=None))
</code></pre>

<p>Now I have another error:</p>

<blockquote>
  <p>UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 in labels with no predicted samples.</p>
</blockquote>