<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Lecture 3
Step1:  We'll train a logistic regression model of the form 
$$
p(y = 1 ~|~ {\bf x}; {\bf w}) = \frac{1}{1 + \textrm{exp}[-(w_0 + w_1x_1 + w_2x_2)]}
$$
using sklearn's logistic regression classifier as follows
Step2:  Q
Step3:  Problem 2
Step4:  Let's also store the documents in a list as follows
Step5:  To be consistent with sklearn conventions, we'll encode the documents as row-vectors stored in a matrix.  In this case, each row of the matrix corresponds to a document, and each column corresponds to a term in the vocabulary.  For our example this gives us a matrix $M$ of shape $3 \times 6$.  The $(d,t)$-entry in $M$ is then the number of times the term $t$ appears in document $d$
Q
Step6:  Hopefully your code returns the matrix 
$$M = 
\left[
\begin{array}{ccccccc}
0 & 0 & 1 & 0 & 1 & 1 \
0 & 0 & 1 & 1 & 0 & 1 \
1 & 1 & 0 & 0 & 1 & 0 \
\end{array}
\right]$$.  
Note that the entry in the (2,0) position is $1$ because the first word (angeles) appears once in the third document. 
OK, let's see how we can construct the same term-frequency matrix in sklearn.  We will use something called the <a href="http
Step7:  The $\texttt{fit_transform}$ method actually does two things.  It fits the model to the training data by building a vocabulary.  It then transforms the text in $D$ into matrix form.  
If we wish to see the vocabulary you can do it like so
Step8:  Note that this is the same vocabulary and indexing that we definfed ourselves.  Hopefully that means we'll get the same term-frequency matrix.  We can print $X$ and check
Step9:  Yep, they're the same!  Notice that we had to convert $X$ to a dense matrix for printing.  This is because CountVectorizer actually returns a sparse matrix.  This is a very good thing since most vectors in a text model will be extremely sparse, since most documents will only contain a handful of words from the vocabulary. 
OK, let's see how we can use the CountVectorizer to transform the test documents into their own term-frequency matrix.  Removing the stop words we have 
OK, now suppose that we have a query document not included in the training set that we want to vectorize.
Step10:  We've already fit the CountVectorizer to the training set, so all we need to do is transform the test set documents into a term-frequency vector using the same conventions.  Since we've already fit the model, we do the transformation with the $\texttt{transform}$ method
Step11:  Let's print it and see what it looks like
Step12:  Notice that the query document included the word $\texttt{new}$ twice, which corresponds to the entry in the $(0,2)$-position. 
Q
Step13:  Let's see what we get when we use sklearn.  Sklearn has a vectorizer called TfidfVectorizer which is similar to CountVectorizer, but it computes tf-idf scores.
Step14:  Note that these are not quite the same, becuase sklearn's implementation of tf-idf uses the add-one smoothing in the denominator for idf. 
<br>
Problem 4
Step15:  The current parameters are set to not remove stop words from the text so that it's a bit easier to explore. 
Look at a few of the reviews stored in $\texttt{text_train}$ as well as their associated labels in $\texttt{labels_train}$.  Can you figure out which label refers to a positive review and which refers to a negative review?
Step16:  The first review is labeled $1$ and has the following text
Step17:  The fourth review is labeled $0$ and has the following text
Step18:  Hopefully it's obvious that label 1 corresponds to positive reviews and label 0 to negative reviews! 
OK, the first thing we'll do is train a logistic regression classifier using the Bag-of-Words model, and see what kind of accuracy we can get.  To get started, we need to vectorize the text into mathematical features that we can use.  We'll use CountVectorizer to do the job. (Before starting, I'm going to reload the data and remove the stop words this time)
Step19:  Q
Step20:  OK, so we got an accuracy of around 81% using Bag-of-Words.  Now lets do the same tests but this time with tf-idf features.
Step21:  WOOHOO!  With tf-idf features we got around 85% accuracy, which is a 4% improvement. (If you're scoffing at this, wait until you get some more experience working with real-world data.  4% improvement is pretty awesome).  
Q
Step22:  We'll train a logistic regression model of the form 
$$
p(y = 1 ~|~ {\bf x}; {\bf w}) = \frac{1}{1 + \textrm{exp}[-(w_0 + w_1x_1 + w_2x_2)]}
$$
using sklearn's logistic regression classifier as follows
Step23:  Q
Step24:  Problem 2
Step25:  Let's also store the documents in a list as follows
Step26:  To be consistent with sklearn conventions, we'll encode the documents as row-vectors stored in a matrix.  In this case, each row of the matrix corresponds to a document, and each column corresponds to a term in the vocabulary.  For our example this gives us a matrix $M$ of shape $3 \times 6$.  The $(d,t)$-entry in $M$ is then the number of times the term $t$ appears in document $d$
Q
Step27:  Hopefully your code returns the matrix 
$$M = 
\left[
\begin{array}{ccccccc}
0 & 0 & 1 & 0 & 1 & 1 \
0 & 0 & 1 & 1 & 0 & 1 \
1 & 1 & 0 & 0 & 1 & 0 \
\end{array}
\right]$$.  
Note that the entry in the (2,0) position is $1$ because the first word (angeles) appears once in the third document. 
OK, let's see how we can construct the same term-frequency matrix in sklearn.  We will use something called the <a href="http
Step28:  The $\texttt{fit_transform}$ method actually does two things.  It fits the model to the training data by building a vocabulary.  It then transforms the text in $D$ into matrix form.  
If we wish to see the vocabulary you can do it like so
Step29:  Note that this is the same vocabulary and indexing that we definfed ourselves.  Hopefully that means we'll get the same term-frequency matrix.  We can print $X$ and check
Step30:  Yep, they're the same!  Notice that we had to convert $X$ to a dense matrix for printing.  This is because CountVectorizer actually returns a sparse matrix.  This is a very good thing since most vectors in a text model will be extremely sparse, since most documents will only contain a handful of words from the vocabulary. 
OK, now suppose that we have a query document not included in the training set that we want to vectorize.
Step31:  We've already fit the CountVectorizer to the training set, so all we need to do is transform the test set documents into a term-frequency vector using the same conventions.  Since we've already fit the model, we do the transformation with the $\texttt{transform}$ method
Step32:  Let's print it and see what it looks like
Step33:  Notice that the query document included the word $\texttt{new}$ twice, which corresponds to the entry in the $(0,2)$-position. 
Q
Step34:  Hopefully you got something like the following
Step35:  Let's see what we get when we use sklearn.  Sklearn has a vectorizer called TfidfVectorizer which is similar to CountVectorizer, but it computes tf-idf scores.
Step36:  Note that these are not quite the same, becuase sklearn's implementation of tf-idf uses the add-one smoothing in the denominator for idf. 
<br>
Problem 4
Step37:  The current parameters are set to not remove stop words from the text so that it's a bit easier to explore. 
Q
Step38:  The first review is labeled $1$ and has the following text
Step39:  The fourth review is labeled $0$ and has the following text
Step40:  Hopefully it's obvious that label 1 corresponds to positive reviews and label 0 to negative reviews! 
OK, the first thing we'll do is train a logistic regression classifier using the Bag-of-Words model, and see what kind of accuracy we can get.  To get started, we need to vectorize the text into mathematical features that we can use.  We'll use CountVectorizer to do the job. (Before starting, I'm going to reload the data and remove the stop words this time)
Step41:  Q
Step42:  A
Step43:  OK, so we got an accuracy of around 81% using Bag-of-Words.  Now lets do the same tests but this time with tf-idf features.
Step45:  WOOHOO!  With tf-idf features we got around 85% accuracy, which is a 4% improvement. (If you're scoffing at this, wait until you get some more experience working with real-world data.  4% improvement is pretty awesome).  
Q
<ASSISTANT_TASK:>
Python Code:
import matplotlib.pyplot as plt
%matplotlib inline 
from sklearn import datasets
iris = datasets.load_iris()
X_train = iris.data[iris.target != 2, :2] # first two features and
y_train = iris.target[iris.target != 2]   # first two labels only 
fig = plt.figure(figsize=(8,8))
mycolors = {"blue": "steelblue", "red": "#a76c6e", "green": "#6a9373"}
plt.scatter(X_train[:, 0], X_train[:, 1], s=100, alpha=0.9, c=[mycolors["red"] if yi==1 else mycolors["blue"] for yi in y_train])
plt.xlabel('sepal length', fontsize=16)
plt.ylabel('sepal width', fontsize=16);
Explanation: Lecture 3: Logistic Regression and Text Models
<img src="figs/logregwordcloud.png",width=1000,height=50>
Problem 1: Logistic Regression for 2D Continuous Features
In the video lecture you saw some examples of using logistic regression to do binary classification on text data (SPAM vs HAM) and on 1D continuous data.  In this problem we'll look at logistic regression for 2D continuous data. The data we'll use are <a href="https://www.math.umd.edu/~petersd/666/html/iris_with_labels.jpg">sepalwikipedia/commons/4/41/Iris_versicolor_3.jpg",width=300,height=50> measurements from the ubiquitous iris dataset.  
<!---
<img style="float:left; width:450px" src="https://upload.wikimedia.org/wikipedia/commons/9/9f/Iris_virginica.jpg",width=300,height=50>
-->
<img style="float:left; width:450px" src="http://www.twofrog.com/images/iris38a.jpg",width=300,height=50>
<!---
<img style="float:right; width:490px" src="https://upload.wikimedia.org/wikipedia/commons/4/41/Iris_versicolor_3.jpg",width=300,height=50>
-->
<img style="float:right; width:490px" src="http://blazingstargardens.com/wp-content/uploads/2016/02/Iris-versicolor-Blue-Flag-Iris1.jpg",width=300,height=62>
The two features of our model will be the sepal length and sepal width.  Execute the following cell to see a plot of the data. The blue points correspond to the sepal measurements of the Iris Setosa (left) and the red points correspond to the sepal measurements of the Iris Versicolour (right).
End of explanation
from sklearn.linear_model import LogisticRegression # import from sklearn 
logreg = LogisticRegression()                       # initialize classifier 
logreg.fit(X_train, y_train);                       # train on training data 
Explanation: We'll train a logistic regression model of the form 
$$
p(y = 1 ~|~ {\bf x}; {\bf w}) = \frac{1}{1 + \textrm{exp}[-(w_0 + w_1x_1 + w_2x_2)]}
$$
using sklearn's logistic regression classifier as follows
End of explanation
import numpy as np
fig = plt.figure(figsize=(8,8))
plt.scatter(X_train[:, 0], X_train[:, 1], s=100, c=[mycolors["red"] if yi==1 else mycolors["blue"] for yi in y_train])
plt.xlabel('Sepal length')
plt.ylabel('Sepal width')
x_min, x_max = np.min(X_train[:,0])-0.1, np.max(X_train[:,0])+0.1
y_min, y_max = np.min(X_train[:,1])-0.1, np.max(X_train[:,1])+0.1
plt.xlim(x_min, x_max)
plt.ylim(y_min, y_max)
x1 = np.linspace(x_min, x_max, 100)
w0 = logreg.intercept_
w1 = logreg.coef_[0][0]
w2 = logreg.coef_[0][1]
x2 = #TODO 
plt.plot(x1, x2, color="gray");
Explanation: Q: Determine the parameters ${\bf w}$ fit by the model.  It might be helpful to consult the documentation for the classifier on the <a href="http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html">sklearn website</a>. Hint: The classifier stores the coefficients and bias term separately. 
Q: In general, what does the Logistic Regression decision boundary look like for data with two features?  
Q: Modify the code below to plot the decision boundary along with the data.
End of explanation
V = {"angeles": 0, "los": 1, "new": 2, "post": 3, "times": 4, "york": 5}
Explanation: Problem 2: The Bag-of-Words Text Model
The remainder of today's exercise will consider the problem of predicting the semantics of text.  In particular, later we'll look at predicting whether movie reviews are positive or negative just based on their text. 
Before we can utilize text as features in a learning model, we need a concise mathematical way to represent things like words, phrases, sentences, etc.  The most common text models are based on the so-called <a href="https://en.wikipedia.org/wiki/Vector_space_model">Vector Space Model</a> (VSM) where individual words in a document are associated with entries of a vector: 
$$
\textrm{"The sky is blue"} \quad \Rightarrow \quad 
\left[
\begin{array}{c}
0 \
1 \ 
0 \
0 \
1
\end{array}
\right]
$$
The first step in creating a VSM is to define a vocabulary, $V$, of words that you will include in your model.  This vocabulary can be determined by looking at all (or most) of the words in the training set, or even by including a fixed vocabulary based on the english language.  A vector representation of a document like a movie review is then a vector with length $|V|$ where each entry in the vector maps uniquely to a word in the vocabulary. A vector encoding of a document would then be a vector that is nonzero in positions corresponding to words present in the document and zero everywhere else.  How you fill in the nonzero entries depends on the model you're using.  Two simple conventions are the Bag-of-Words model and the binary model.  
In the binary model we simply set an entry of the vector to $1$ if the associate word appears at least once in the document.  In the more common Bag-of-Words model we set an entry of the vector equal to the frequency with which the word appears in the document. Let's see if we can come up with a simple implementation of the Bag-of-Words model in Python, and then later we'll see how sklearn can do the heavy lifting for us. 
Consider a training set containing three documents, specified as follows 
$\texttt{Training Set}:$
$\texttt{d1}: \texttt{new york times}$
$\texttt{d2}: \texttt{new york post}$
$\texttt{d3}: \texttt{los angeles times}$
First we'll define the vocabulary based on the words in the test set.  It is $V = { \texttt{angeles}, \texttt{los}, \texttt{new}, \texttt{post}, \texttt{times}, \texttt{york}}$. 
We need to define an association between the particular words in the vocabulary and the specific entries in our vectors.  Let's define this association in the order that we've listed them above.  We can store this mapping as a Python dictionary as follows:
End of explanation
D = ["new york times", "new york post", "los angeles times"]
Explanation: Let's also store the documents in a list as follows:
End of explanation
M = np.zeros((len(D),len(V)))
for ii, doc in enumerate(D): 
    for term in doc.split(): 
        #TODO 
        
print M
Explanation: To be consistent with sklearn conventions, we'll encode the documents as row-vectors stored in a matrix.  In this case, each row of the matrix corresponds to a document, and each column corresponds to a term in the vocabulary.  For our example this gives us a matrix $M$ of shape $3 \times 6$.  The $(d,t)$-entry in $M$ is then the number of times the term $t$ appears in document $d$
Q: Your first task is to write some simple Python code to construct the term-frequency matrix $M$
End of explanation
from sklearn.feature_extraction.text import CountVectorizer    # import CountVectorizer 
vectorizer = CountVectorizer()                                 # initialize the vectorizer
X = vectorizer.fit_transform(D)                                # fit to training data and transform to matrix 
Explanation: Hopefully your code returns the matrix 
$$M = 
\left[
\begin{array}{ccccccc}
0 & 0 & 1 & 0 & 1 & 1 \
0 & 0 & 1 & 1 & 0 & 1 \
1 & 1 & 0 & 0 & 1 & 0 \
\end{array}
\right]$$.  
Note that the entry in the (2,0) position is $1$ because the first word (angeles) appears once in the third document. 
OK, let's see how we can construct the same term-frequency matrix in sklearn.  We will use something called the <a href="http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html">CountVectorizer</a> to accomplish this. Let's see some code and then we'll explain how it functions.
End of explanation
print vectorizer.vocabulary_
Explanation: The $\texttt{fit_transform}$ method actually does two things.  It fits the model to the training data by building a vocabulary.  It then transforms the text in $D$ into matrix form.  
If we wish to see the vocabulary you can do it like so
End of explanation
print X.todense()
Explanation: Note that this is the same vocabulary and indexing that we definfed ourselves.  Hopefully that means we'll get the same term-frequency matrix.  We can print $X$ and check
End of explanation
d4 = ["new york new tribune"]
Explanation: Yep, they're the same!  Notice that we had to convert $X$ to a dense matrix for printing.  This is because CountVectorizer actually returns a sparse matrix.  This is a very good thing since most vectors in a text model will be extremely sparse, since most documents will only contain a handful of words from the vocabulary. 
OK, let's see how we can use the CountVectorizer to transform the test documents into their own term-frequency matrix.  Removing the stop words we have 
OK, now suppose that we have a query document not included in the training set that we want to vectorize.
End of explanation
x4 = vectorizer.transform(d4)
Explanation: We've already fit the CountVectorizer to the training set, so all we need to do is transform the test set documents into a term-frequency vector using the same conventions.  Since we've already fit the model, we do the transformation with the $\texttt{transform}$ method:
End of explanation
print x4.todense()
Explanation: Let's print it and see what it looks like
End of explanation
row_norms = np.array([np.linalg.norm(row) for row in Xtfidf])
X_tfidf_n = np.dot(np.diag(1./row_norms), Xtfidf)
print X_tfidf_n
Explanation: Notice that the query document included the word $\texttt{new}$ twice, which corresponds to the entry in the $(0,2)$-position. 
Q: What's missing from $x4$ that we might expect to see from the query document? 
<br>
Problem 3: Term Frequency - Inverse Document Frequency
The Bag-of-Words model for text classification is very popular, but let's see if we can do better.  Currently we're weighting every word in the corpus by it's frequency.  It turns out that in text classification there are often features that are not particularly useful predictors for the document class, either because they are too common or too uncommon.  Stop-words are extremely common, low-information words like "a", "the", "as", etc.  Removing these from documents is typically the first thing done in peparing data for document classification. 
Q: Can you think of a situation where it might be useful to keep stop words in the corpus? 
Other words that tend to be uninformative predictors are words that appear very very rarely.  In particular, if they do not appear frequently enough in the training data then it is difficult for a classification algorithm to weight them heavily in the classification process. 
In general, the words that tend to be useful predictors are the words that appear frequently, but not too frequently.  Consider the following frequency graph for a corpus. 
<img src="figs/feat_freq.png",width=400,height=50>
The features in column A appear too frequently to be very useful, and the features in column C appear too rarely. One first-pass method of feature selection in text classification would be to discard the words from columns A and C, and build a classifier with only features from column B.
Another common model for identifying the useful terms in a document is the Term Frequency - Inverse Document Frequency (tf-idf) model.  Here we won't throw away any terms, but we'll replace their Bag-of-Words frequency counts with tf-idf scores which we describe below. 
The tf-idf score is the product of two statistics, term frequency and inverse document frequency
$$\texttt{tfidf(d,t)} = \texttt{tf(d,t)} \times \texttt{idf(t)}$$
The term frequency $\texttt{tf(d,t)}$ is a measure of the frequency with which term $t$ appears in document $d$.  The inverse document frequency $\texttt{idf(t)}$ is a measure of how much information the word provides, that is, whether the term is common or rare across all documents.  By multiplying the two quantities together, we obtain a representation of term $t$ in document $d$ that weighs how common the term is in the document with how common the word is in the entire corpus. You can imagine that the words that get the highest associated values are terms that appear many times in a small number of documents. 
There are many ways to compute the composite terms $\texttt{tf}$ and $\texttt{idf}$.  For simplicity, we'll define $\texttt{tf(d,t)}$ to be the number of times term $t$ appears in document $d$ (i.e., Bag-of-Words). We will define the inverse document frequency as follows: 
$$
\texttt{idf(t)} = \ln ~ \frac{\textrm{total # documents}}{\textrm{1 + # documents with term }t}
 = \ln ~ \frac{|D|}{|d: ~ t \in d |}
$$
Note that we could have a potential problem if a term comes up that is not in any of the training documents, resulting  in a divide by zero.  This might happen if you use a canned vocabulary instead of constructing one from the training documents.  To guard against this, many implementations will use add-one smoothing in the denominator (this is what sklearn does). 
$$
\texttt{idf(t)} = \ln ~ \frac{\textrm{total # documents}}{\textrm{1 + # documents with term }t}
 = \ln ~ \frac{|D|}{1 + |d: ~ t \in d |}
$$
Q: Compute $\texttt{idf(t)}$ (without smoothing) for each of the terms in the training documents from the previous problem
Q: Compute the td-ifd matrix for the training set 
Hopefully you got something like the following: 
$$
X_{tfidf} = 
\left[
\begin{array}{ccccccccc}
0.          & 0.         &  0.40546511 &  0.         &  0.40546511 &  0.40546511 \
0.          & 0.         &  0.40546511 &  1.09861229 &  0.         &  0.40546511 \
1.09861229  & 1.09861229 &  0.         &  0.         &  0.40546511 &  0.      
\end{array}
\right]
$$
The final step in any VSM method is the normalization of the vectors.  This is done so that very long documents to not completely overpower the small and medium length documents.
End of explanation
from sklearn.feature_extraction.text import TfidfVectorizer
tfidf = TfidfVectorizer()
Y = tfidf.fit_transform(D)
print Y.todense()
Explanation: Let's see what we get when we use sklearn.  Sklearn has a vectorizer called TfidfVectorizer which is similar to CountVectorizer, but it computes tf-idf scores.
End of explanation
import csv 
def read_and_clean_data(fname, remove_stops=True):
    
    with open('data/stopwords.txt', 'rt') as f:
        stops = [line.rstrip('\n') for line in f]
    
    with open(fname,'rt') as tsvin:
        reader = csv.reader(tsvin, delimiter='\t')
        labels = []; text = [] 
        for ii, row in enumerate(reader):
            labels.append(int(row[0]))
            words = row[1].lower().split()
            words = [w for w in words if not w in stops] if remove_stops else words 
            text.append(" ".join(words))
    
    return text, labels
text_train, labels_train = read_and_clean_data('data/labeledTrainData.tsv', remove_stops=True)
text_test, labels_test = read_and_clean_data('data/labeledTestData.tsv', remove_stops=True)
Explanation: Note that these are not quite the same, becuase sklearn's implementation of tf-idf uses the add-one smoothing in the denominator for idf. 
<br>
Problem 4: Classifying Semantics in Movie Reviews
The data for this problem was taken from the <a href="https://www.kaggle.com/c/word2vec-nlp-tutorial/details/part-1-for-beginners-bag-of-words">Bag of Words Meets Bag of Popcorn</a> Kaggle competition
In this problem you will use the text from movie reviews to predict whether the reviewer felt positively or negatively about the movie using Bag-of-Words and tf-idf. I've partially cleaned the data and stored it in files called $\texttt{labeledTrainData.tsv}$ and $\texttt{labeledTestData.tsv}$ in the data directory.
End of explanation
labels_train[:4]
Explanation: The current parameters are set to not remove stop words from the text so that it's a bit easier to explore. 
Look at a few of the reviews stored in $\texttt{text_train}$ as well as their associated labels in $\texttt{labels_train}$.  Can you figure out which label refers to a positive review and which refers to a negative review?
End of explanation
text_train[3]
Explanation: The first review is labeled $1$ and has the following text:
End of explanation
text_train[0]
Explanation: The fourth review is labeled $0$ and has the following text:
End of explanation
text_train, labels_train = read_and_clean_data('data/labeledTrainData.tsv', remove_stops=True)
text_test, labels_test = read_and_clean_data('data/labeledTestData.tsv', remove_stops=True)
cvec = CountVectorizer()
X_bw_train = cvec.fit_transform(text_train)
y_train = np.array(labels_train)
X_bw_test  = cvec.transform(text_test)
y_test  = np.array(labels_test)
Explanation: Hopefully it's obvious that label 1 corresponds to positive reviews and label 0 to negative reviews! 
OK, the first thing we'll do is train a logistic regression classifier using the Bag-of-Words model, and see what kind of accuracy we can get.  To get started, we need to vectorize the text into mathematical features that we can use.  We'll use CountVectorizer to do the job. (Before starting, I'm going to reload the data and remove the stop words this time)
End of explanation
from sklearn.metrics import accuracy_score
bwLR = LogisticRegression()
bwLR.fit(X_bw_train, y_train)
pred_bwLR = bwLR.predict(X_bw_test)
print "Logistic Regression accuracy with Bag-of-Words: ", accuracy_score(y_test, pred_bwLR)
Explanation: Q: How many different words are in the vocabulary? 
OK, now we'll train a logistic regression classifier on the training set, and test the accuracy on the test set.  To do this we'll need to load some kind of accuracy metric from sklearn.
End of explanation
tvec = TfidfVectorizer()
X_tf_train = tvec.fit_transform(text_train)
X_tf_test  = tvec.transform(text_test)
tfLR = LogisticRegression()
tfLR.fit(X_tf_train, y_train)
pred_tfLR = tfLR.predict(X_tf_test)
print "Logistic Regression accuracy with tf-idf: ", accuracy_score(y_test, pred_tfLR)
Explanation: OK, so we got an accuracy of around 81% using Bag-of-Words.  Now lets do the same tests but this time with tf-idf features.
End of explanation
import matplotlib.pyplot as plt
%matplotlib inline 
from sklearn import datasets
iris = datasets.load_iris()
X_train = iris.data[iris.target != 2, :2] # first two features and
y_train = iris.target[iris.target != 2]   # first two labels only 
fig = plt.figure(figsize=(8,8))
mycolors = {"blue": "steelblue", "red": "#a76c6e", "green": "#6a9373"}
plt.scatter(X_train[:, 0], X_train[:, 1], s=100, alpha=0.9, c=[mycolors["red"] if yi==1 else mycolors["blue"] for yi in y_train])
plt.xlabel('sepal length', fontsize=16)
plt.ylabel('sepal width', fontsize=16);
Explanation: WOOHOO!  With tf-idf features we got around 85% accuracy, which is a 4% improvement. (If you're scoffing at this, wait until you get some more experience working with real-world data.  4% improvement is pretty awesome).  
Q: Which words are the strongest predictors for a positive review and which words are the strongest predictors for negative reviews? I'm not going to give you the answer to this one because it's the same question we'll ask on the next homework assignment. But if you figure this out you'll have a great head start! 
<br><br><br>
<br><br><br>
<br><br><br>
<br><br><br>
<br><br><br>
<br><br><br>
<br><br><br>
<br><br><br>
<br><br><br>
<br><br><br>
<br><br><br>
<br><br><br>
<br><br><br>
<br><br><br>
<br><br><br>
<br><br><br>
Notebook Solutions
<br><br><br>
Problem 1: Logistic Regression for 2D Continuous Features
In the video lecture you saw some examples of using logistic regression to do binary classification on text data (SPAM vs HAM) and on 1D continuous data.  In this problem we'll look at logistic regression for 2D continuous data. The data we'll use are <a href="https://www.math.umd.edu/~petersd/666/html/iris_with_labels.jpg">sepal</a> measurements from the ubiquitous iris dataset.  
<!---
<img style="float:left; width:450px" src="https://upload.wikimedia.org/wikipedia/commons/9/9f/Iris_virginica.jpg",width=300,height=50>
-->
<img style="float:left; width:450px" src="http://www.twofrog.com/images/iris38a.jpg",width=300,height=50>
<!---
<img style="float:right; width:490px" src="https://upload.wikimedia.org/wikipedia/commons/4/41/Iris_versicolor_3.jpg",width=300,height=50>
-->
<img style="float:right; width:490px" src="http://blazingstargardens.com/wp-content/uploads/2016/02/Iris-versicolor-Blue-Flag-Iris1.jpg",width=300,height=62>
The two features of our model will be the sepal length and sepal width.  Execute the following cell to see a plot of the data. The blue points correspond to the sepal measurements of the Iris Setosa (left) and the red points correspond to the sepal measurements of the Iris Versicolour (right).
End of explanation
from sklearn.linear_model import LogisticRegression # import from sklearn 
logreg = LogisticRegression()                       # initialize classifier 
logreg.fit(X_train, y_train);                       # train on training data 
Explanation: We'll train a logistic regression model of the form 
$$
p(y = 1 ~|~ {\bf x}; {\bf w}) = \frac{1}{1 + \textrm{exp}[-(w_0 + w_1x_1 + w_2x_2)]}
$$
using sklearn's logistic regression classifier as follows
End of explanation
import numpy as np
fig = plt.figure(figsize=(8,8))
plt.scatter(X_train[:, 0], X_train[:, 1], s=100, c=[mycolors["red"] if yi==1 else mycolors["blue"] for yi in y_train])
plt.xlabel('Sepal length')
plt.ylabel('Sepal width')
x_min, x_max = np.min(X_train[:,0])-0.1, np.max(X_train[:,0])+0.1
y_min, y_max = np.min(X_train[:,1])-0.1, np.max(X_train[:,1])+0.1
plt.xlim(x_min, x_max)
plt.ylim(y_min, y_max)
x1 = np.linspace(x_min, x_max, 100)
w0 = logreg.intercept_
w1 = logreg.coef_[0][0]
w2 = logreg.coef_[0][1]
x2 = -(w0/w2) - (w1/w2)*x1 #TODO 
plt.plot(x1, x2, color="gray");
Explanation: Q: Determine the parameters ${\bf w}$ fit by the model.  It might be helpful to consult the documentation for the classifier on the <a href="http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html">sklearn website</a>. Hint: The classifier stores the coefficients and bias term separately. 
A: The bias term is stored in logreg.intercept_ . The remaining coefficients are stored in logreg.coef_ . For this problem we have 
$$
w_0 =-0.599, \quad w_1 = 2.217, \quad \textrm{and} \quad w_2 =  -3.692
$$
Q: In general, what does the Logistic Regression decision boundary look like for data with two features?  
A: The decision boundary for Logistic Regresion for data with two features is a line.  To see this, remember that the decision boundary is made up of $(x_1, x_2)$ points such that $\textrm{sigm}({\bf w}^T{\bf x}) = 0.5$.  We then have 
$$
\frac{1}{1 + \textrm{exp}[-(w_0 + w_1x_1 + w_2x_2)]} = \frac{1}{2} ~~\Rightarrow ~~ w_0 + w_1x_1 + w_2x_2 = 0 ~~\Rightarrow~~ x_2 = -\frac{w_1}{w_2}x_1 - \frac{w_0}{w_2}
$$
So the decision boundary is a line with slope $-w_1/w_2$ and intercept $-w_0/w_2$. 
Q: Modify the code below to plot the decision boundary along with the data.
End of explanation
V = {"angeles": 0, "los": 1, "new": 2, "post": 3, "times": 4, "york": 5}
Explanation: Problem 2: The Bag-of-Words Text Model
The remainder of today's exercise will consider the problem of predicting the semantics of text.  In particular, later we'll look at predicting whether movie reviews are positive or negative just based on their text. 
Before we can utilize text as features in a learning model, we need a concise mathematical way to represent things like words, phrases, sentences, etc.  The most common text models are based on the so-called <a href="https://en.wikipedia.org/wiki/Vector_space_model">Vector Space Model</a> (VSM) where individual words in a document are associated with entries of a vector: 
$$
\textrm{"The sky is blue"} \quad \Rightarrow \quad 
\left[
\begin{array}{c}
0 \
1 \ 
0 \
0 \
1
\end{array}
\right]
$$
The first step in creating a VSM is to define a vocabulary, $V$, of words that you will include in your model.  This vocabulary can be determined by looking at all (or most) of the words in the training set, or even by including a fixed vocabulary based on the english language.  A vector representation of a document like a movie review is then a vector with length $|V|$ where each entry in the vector maps uniquely to a word in the vocabulary. A vector encoding of a document would then be a vector that is nonzero in positions corresponding to words present in the document and zero everywhere else.  How you fill in the nonzero entries depends on the model you're using.  Two simple conventions are the Bag-of-Words model and the binary model.  
In the binary model we simply set an entry of the vector to $1$ if the associate word appears at least once in the document.  In the more common Bag-of-Words model we set an entry of the vector equal to the frequency with which the word appears in the document. Let's see if we can come up with a simple implementation of the Bag-of-Words model in Python, and then later we'll see how sklearn can do the heavy lifting for us. 
Consider a training set containing three documents, specified as follows 
$\texttt{Training Set}:$
$\texttt{d1}: \texttt{new york times}$
$\texttt{d2}: \texttt{new york post}$
$\texttt{d3}: \texttt{los angeles times}$
First we'll define the vocabulary based on the words in the test set.  It is $V = { \texttt{angeles}, \texttt{los}, \texttt{new}, \texttt{post}, \texttt{times}, \texttt{york}}$. 
We need to define an association between the particular words in the vocabulary and the specific entries in our vectors.  Let's define this association in the order that we've listed them above.  We can store this mapping as a Python dictionary as follows:
End of explanation
D = ["new york times", "new york post", "los angeles times"]
Explanation: Let's also store the documents in a list as follows:
End of explanation
M = np.zeros((len(D),len(V)))
for ii, doc in enumerate(D): 
    for term in doc.split(): 
        M[ii, V[term]] += 1
        
print M
Explanation: To be consistent with sklearn conventions, we'll encode the documents as row-vectors stored in a matrix.  In this case, each row of the matrix corresponds to a document, and each column corresponds to a term in the vocabulary.  For our example this gives us a matrix $M$ of shape $3 \times 6$.  The $(d,t)$-entry in $M$ is then the number of times the term $t$ appears in document $d$
Q: Your first task is to write some simple Python code to construct the term-frequency matrix $M$
End of explanation
from sklearn.feature_extraction.text import CountVectorizer    # import CountVectorizer 
vectorizer = CountVectorizer()                                 # initialize the vectorizer
X = vectorizer.fit_transform(D)                                # fit to training data and transform to matrix 
Explanation: Hopefully your code returns the matrix 
$$M = 
\left[
\begin{array}{ccccccc}
0 & 0 & 1 & 0 & 1 & 1 \
0 & 0 & 1 & 1 & 0 & 1 \
1 & 1 & 0 & 0 & 1 & 0 \
\end{array}
\right]$$.  
Note that the entry in the (2,0) position is $1$ because the first word (angeles) appears once in the third document. 
OK, let's see how we can construct the same term-frequency matrix in sklearn.  We will use something called the <a href="http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html">CountVectorizer</a> to accomplish this. Let's see some code and then we'll explain how it functions.
End of explanation
print vectorizer.vocabulary_
Explanation: The $\texttt{fit_transform}$ method actually does two things.  It fits the model to the training data by building a vocabulary.  It then transforms the text in $D$ into matrix form.  
If we wish to see the vocabulary you can do it like so
End of explanation
print X.todense()
Explanation: Note that this is the same vocabulary and indexing that we definfed ourselves.  Hopefully that means we'll get the same term-frequency matrix.  We can print $X$ and check
End of explanation
d4 = ["new york new tribune"]
Explanation: Yep, they're the same!  Notice that we had to convert $X$ to a dense matrix for printing.  This is because CountVectorizer actually returns a sparse matrix.  This is a very good thing since most vectors in a text model will be extremely sparse, since most documents will only contain a handful of words from the vocabulary. 
OK, now suppose that we have a query document not included in the training set that we want to vectorize.
End of explanation
x4 = vectorizer.transform(d4)
Explanation: We've already fit the CountVectorizer to the training set, so all we need to do is transform the test set documents into a term-frequency vector using the same conventions.  Since we've already fit the model, we do the transformation with the $\texttt{transform}$ method:
End of explanation
print x4.todense()
Explanation: Let's print it and see what it looks like
End of explanation
idf = np.array([np.log(3), np.log(3), np.log(3./2), np.log(3), np.log(3./2), np.log(3./2)])
Xtfidf = np.dot(X.todense(), np.diag(idf))
print Xtfidf
Explanation: Notice that the query document included the word $\texttt{new}$ twice, which corresponds to the entry in the $(0,2)$-position. 
Q: What's missing from $x4$ that we might expect to see from the query document? 
A: The word $\texttt{tribune}$ do not appear in vector $x4$ at all.  This is because it did not occur in the training set, which means it is not present in the VSM vocabulary.  This should not bother us too much.  Most reasonable text data sets will have most of the important words present in the training set and thus in the vocabulary.  On the other hand, the throw-away words that are present only in the test set are probably useless anyway, since the learning model is trained based on the text in the training set, and thus won't be able to do anything intelligent with words the model hasn't seen yet. 
<br>
Problem 3: Term Frequency - Inverse Document Frequency
The Bag-of-Words model for text classification is very popular, but let's see if we can do better.  Currently we're weighting every word in the corpus by it's frequency.  It turns out that in text classification there are often features that are not particularly useful predictors for the document class, either because they are too common or too uncommon.  Stop-words are extremely common, low-information words like "a", "the", "as", etc.  Removing these from documents is typically the first thing done in peparing data for document classification. 
Q: Can you think of a situation where it might be useful to keep stop words in the corpus? 
A: If you plan to use bi-grams or tri-grams as features.  Bi-grams are pairs of words that appear side-by-side in a document, e.g. "he went", "went to", "to the", "the store". 
Other words that tend to be uninformative predictors are words that appear very very rarely.  In particular, if they do not appear frequently enough in the training data then it is difficult for a classification algorithm to weight them heavily in the classification process. 
In general, the words that tend to be useful predictors are the words that appear frequently, but not too frequently.  Consider the following frequency graph for a corpus. 
<img src="figs/feat_freq.png",width=400,height=50>
The features in column A appear too frequently to be very useful, and the features in column C appear too rarely. One first-pass method of feature selection in text classification would be to discard the words from columns A and C, and build a classifier with only features from column B.
Another common model for identifying the useful terms in a document is the Term Frequency - Inverse Document Frequency (tf-idf) model.  Here we won't throw away any terms, but we'll replace their Bag-of-Words frequency counts with tf-idf scores which we describe below. 
The tf-idf score is the product of two statistics, term frequency and inverse document frequency
$$\texttt{tfidf(d,t)} = \texttt{tf(d,t)} \times \texttt{idf(t)}$$
The term frequency $\texttt{tf(d,t)}$ is a measure of the frequency with which term $t$ appears in document $d$.  The inverse document frequency $\texttt{idf(t)}$ is a measure of how much information the word provides, that is, whether the term is common or rare across all documents.  By multiplying the two quantities together, we obtain a representation of term $t$ in document $d$ that weighs how common the term is in the document with how common the word is in the entire corpus. You can imagine that the words that get the highest associated values are terms that appear many times in a small number of documents. 
There are many ways to compute the composite terms $\texttt{tf}$ and $\texttt{idf}$.  For simplicity, we'll define $\texttt{tf(d,t)}$ to be the number of times term $t$ appears in document $d$ (i.e., Bag-of-Words). We will define the inverse document frequency as follows: 
$$
\texttt{idf(t)} = \ln ~ \frac{\textrm{total # documents}}{\textrm{# documents with term }t}
 = \ln ~ \frac{|D|}{|d: ~ t \in d |}
$$
Note that we could have a potential problem if a term comes up that is not in any of the training documents, resulting  in a divide by zero.  This might happen if you use a canned vocabulary instead of constructing one from the training documents.  To guard against this, many implementations will use add-one smoothing in the denominator (this is what sklearn does). 
$$
\texttt{idf(t)} = \ln ~ \frac{\textrm{total # documents}}{\textrm{1 + # documents with term }t}
 = \ln ~ \frac{|D|}{1 + |d: ~ t \in d |}
$$
Q: Compute $\texttt{idf(t)}$ (without smoothing) for each of the terms in the training documents from the previous problem
A: 
$
\texttt{idf}(\texttt{angeles}) = \ln ~ \frac{3}{1} = \ln ~ \frac{3}{1} = 1.10
$
$
\texttt{idf}(\texttt{los}) = \ln ~ \frac{3}{1} = \ln ~ \frac{3}{1} = 1.10
$
$
\texttt{idf}(\texttt{new}) = \ln ~ \frac{3}{2} = \ln ~ \frac{3}{2} = 0.41
$
$
\texttt{idf}(\texttt{post}) = \ln ~ \frac{3}{1} = \ln ~ \frac{3}{1} = 1.10
$
$
\texttt{idf}(\texttt{times}) = \ln ~ \frac{3}{2} = \ln ~ \frac{3}{2} = 0.41
$
$
\texttt{idf}(\texttt{york}) = \ln ~ \frac{3}{2} = \ln ~ \frac{3}{2} = 0.41
$
Q: Compute the td-ifd matrix for the training set 
A: There are several ways to do this.  One way would be to multiply the term-frequency matrix on the right with a diagonal matrix with the idf-values on the main diagonal
End of explanation
row_norms = np.array([np.linalg.norm(row) for row in Xtfidf])
X_tfidf_n = np.dot(np.diag(1./row_norms), Xtfidf)
print X_tfidf_n
Explanation: Hopefully you got something like the following: 
$$
X_{tfidf} = 
\left[
\begin{array}{ccccccccc}
0.          & 0.         &  0.40546511 &  0.         &  0.40546511 &  0.40546511 \
0.          & 0.         &  0.40546511 &  1.09861229 &  0.         &  0.40546511 \
1.09861229  & 1.09861229 &  0.         &  0.         &  0.40546511 &  0.      
\end{array}
\right]
$$
The final step in any VSM method is the normalization of the vectors.  This is done so that very long documents to not completely overpower the small and medium length documents.
End of explanation
from sklearn.feature_extraction.text import TfidfVectorizer
tfidf = TfidfVectorizer()
Y = tfidf.fit_transform(D)
print Y.todense()
Explanation: Let's see what we get when we use sklearn.  Sklearn has a vectorizer called TfidfVectorizer which is similar to CountVectorizer, but it computes tf-idf scores.
End of explanation
import csv 
def read_and_clean_data(fname, remove_stops=True):
    
    with open('data/stopwords.txt', 'r') as f:
        stops = [line.rstrip('\n') for line in f]
    
    with open(fname,'rb') as tsvin:
        reader = csv.reader(tsvin, delimiter='\t')
        labels = []; text = [] 
        for ii, row in enumerate(reader):
            labels.append(int(row[0]))
            words = row[1].lower().split()
            words = [w for w in words if not w in stops] if remove_stops else words 
            text.append(" ".join(words))
    
    return text, labels
text_train, labels_train = read_and_clean_data('data/labeledTrainData.tsv', remove_stops=False)
text_test, labels_test = read_and_clean_data('data/labeledTestData.tsv', remove_stops=False)
Explanation: Note that these are not quite the same, becuase sklearn's implementation of tf-idf uses the add-one smoothing in the denominator for idf. 
<br>
Problem 4: Classifying Semantics in Movie Reviews
The data for this problem was taken from the <a href="https://www.kaggle.com/c/word2vec-nlp-tutorial/details/part-1-for-beginners-bag-of-words">Bag of Words Meets Bag of Popcorn</a> Kaggle competition
In this problem you will use the text from movie reviews to predict whether the reviewer felt positively or negatively about the movie using Bag-of-Words and tf-idf. I've partially cleaned the data and stored it in files called $\texttt{labeledTrainData.tsv}$ and $\texttt{labeledTestData.tsv}$ in the data directory.
End of explanation
labels_train[:4]
Explanation: The current parameters are set to not remove stop words from the text so that it's a bit easier to explore. 
Q: Look at a few of the reviews stored in $\texttt{text_train}$ as well as their associated labels in $\texttt{labels_train}$.  Can you figure out which label refers to a positive review and which refers to a negative review? 
A:
End of explanation
text_train[0]
Explanation: The first review is labeled $1$ and has the following text:
End of explanation
text_train[3]
Explanation: The fourth review is labeled $0$ and has the following text:
End of explanation
text_train, labels_train = read_and_clean_data('data/labeledTrainData.tsv', remove_stops=True)
text_test, labels_test = read_and_clean_data('data/labeledTestData.tsv', remove_stops=True)
cvec = CountVectorizer()
X_bw_train = cvec.fit_transform(text_train)
y_train = np.array(labels_train)
X_bw_test  = cvec.transform(text_test)
y_test  = np.array(labels_test)
Explanation: Hopefully it's obvious that label 1 corresponds to positive reviews and label 0 to negative reviews! 
OK, the first thing we'll do is train a logistic regression classifier using the Bag-of-Words model, and see what kind of accuracy we can get.  To get started, we need to vectorize the text into mathematical features that we can use.  We'll use CountVectorizer to do the job. (Before starting, I'm going to reload the data and remove the stop words this time)
End of explanation
X_bw_train.shape
Explanation: Q: How many different words are in the vocabulary?
End of explanation
from sklearn.metrics import accuracy_score
bwLR = LogisticRegression()
bwLR.fit(X_bw_train, y_train)
pred_bwLR = bwLR.predict(X_bw_test)
print "Logistic Regression accuracy with Bag-of-Words: ", accuracy_score(y_test, pred_bwLR)
Explanation: A: It looks like around 17,800 distinct words 
OK, now we'll train a logistic regression classifier on the training set, and test the accuracy on the test set.  To do this we'll need to load some kind of accuracy metric from sklearn.
End of explanation
tvec = TfidfVectorizer()
X_tf_train = tvec.fit_transform(text_train)
X_tf_test  = tvec.transform(text_test)
tfLR = LogisticRegression()
tfLR.fit(X_tf_train, y_train)
pred_tfLR = tfLR.predict(X_tf_test)
print "Logistic Regression accuracy with tf-idf: ", accuracy_score(y_test, pred_tfLR)
Explanation: OK, so we got an accuracy of around 81% using Bag-of-Words.  Now lets do the same tests but this time with tf-idf features.
End of explanation
from IPython.core.display import HTML
HTML(
<style>
.MathJax nobr>span.math>span{border-left-width:0 !important};
</style>
)
Explanation: WOOHOO!  With tf-idf features we got around 85% accuracy, which is a 4% improvement. (If you're scoffing at this, wait until you get some more experience working with real-world data.  4% improvement is pretty awesome).  
Q: Which words are the strongest predictors for a positive review and which words are the strongest predictors for negative reviews? I'm not going to give you the answer to this one because it's the same question we'll ask on the next homework assignment. But if you figure this out you'll have a great head start! 
<br><br><br>
<br><br><br>
<br><br><br>
<br><br><br>
<br><br><br>
End of explanation
<END_TASK>