How is the TFIDFVectorizer in scikit-learn supposed to work?
<p>I'm trying to get words that are distinctive of certain documents using the TfIDFVectorizer class in scikit-learn. It creates a tfidf matrix with all the words and their scores in all the documents, but then it seems to count common words, as well. This is some of the code I'm running: </p>

<pre><code>vectorizer = TfidfVectorizer()
tfidf_matrix = vectorizer.fit_transform(contents)
feature_names = vectorizer.get_feature_names()
dense = tfidf_matrix.todense()
denselist = dense.tolist()
df = pd.DataFrame(denselist, columns=feature_names, index=characters)
s = pd.Series(df.loc['Adam'])
s[s &gt; 0].sort_values(ascending=False)[:10]
</code></pre>

<p>I expected this to return a list of distinctive words for the document 'Adam', but what it does it return a list of common words: </p>

<pre><code>and     0.497077
to      0.387147
the     0.316648
of      0.298724
in      0.186404
with    0.144583
his     0.140998
</code></pre>

<p>I might not understand it perfectly, but as I understand it, tf-idf is supposed to find words that are distinctive of one document in a corpus, finding words that appear frequently in one document, but not in other documents. Here, <code>and</code> appears frequently in other documents, so I don't know why it's returning a high value here. </p>

<p>The complete code I'm using to generate this is <a href="https://github.com/JonathanReeve/milton-analysis/blob/v0.1/tfidf-scikit.ipynb" rel="noreferrer">in this Jupyter notebook</a>. </p>

<p>When I compute tf/idfs semi-manually, using the NLTK and computing scores for each word, I get the appropriate results. For the 'Adam' document: </p>

<pre><code>fresh        0.000813
prime        0.000813
bone         0.000677
relate       0.000677
blame        0.000677
enough       0.000677
</code></pre>

<p>That looks about right, since these are words that appear in the 'Adam' document, but not as much in other documents in the corpus. The complete code used to generate this is in <a href="https://github.com/JonathanReeve/milton-analysis/blob/v0.1/tfidf-nltk.ipynb" rel="noreferrer">this Jupyter notebook</a>. </p>

<p>Am I doing something wrong with the scikit code? Is there another way to initialize this class where it returns the right results? Of course, I can ignore stopwords by passing <code>stop_words = 'english'</code>, but that doesn't really solve the problem, since common words of any sort shouldn't have high scores here. </p>