<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Bag of words meets bag of popcorn
A tutorial in text mining and NLP
Please first download the data from here
Step1:  If you are missing bs4 or nltk you can install them via
Step2:  Let's take a quick look at the data
Step3:  In particular note that the review column has some html tags
Step4:  Our target is to use sentiment column to predict the same for the test set
Step6:  Now we need some sort of "cleaning" processes, we simply eliminate all the non-alphabet characters and use BeautifulSoup library to extract the text content, Let's put everything together in a function
Step7:  Before proceeding, let's test what our function does
Step8:  and with the remove_stopwords flag on, it will give us
Step9:  and with split flag on, it can actually perform a simple tokenization
Step10:  Notice the words
reading, purely, written, 
raised, films, clearly 
that all need stemming, but let's for now continue with what we have
Let's now apply our cleaning process to the review columns
Step11:  Our corpus is all of the reviews
Step12:  Not let's use sklearn's tf-idf vectorizer with unigram and bigrams, and a log TF function (sublinear_tf=True)
Note that we can remove the stop_words here
Step13:  We can now use the object tfv to build the tf-idf vector-space representation of the reviews, the transformation returns a sparse scipy matrix
Note
Step14:  Notice the shape of the X_all matrix
Step15:  So it created about 300K numerical features! (the total count of words in the corpus + number of unique bigrams)
It is highly sparse though (which allows python to use scipy's sparse matrix representation and keep everything on the RAM!)
Now let's split the X_all matrix back to our train and test set
Step16:  We now use a Logistic Regression model to fit to the numerical features, (LR is quite safe here to use for such a high number of features, to use tree based models we definitely need feature selection)
Let's perform a simple 5-fold cross-validation using AUC score and also fine tune one of the parameters of the LR model, the penalty constant c
Step17:  Our CV experiment suggests that c = 30 is the best choice, so we use our best model to fit to the entire train set now
Step18:  and finally predicting for test set and storing the results
<ASSISTANT_TASK:>
Python Code:
import pandas as pd
import numpy as np
import re
from nltk.corpus import stopwords
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.cross_validation import cross_val_score
from os.path import join
from bs4 import BeautifulSoup
Explanation: Bag of words meets bag of popcorn
A tutorial in text mining and NLP
Please first download the data from here:
https://www.kaggle.com/c/word2vec-nlp-tutorial/data
Let's first import all the libraries we will need
End of explanation
root_dir = '/Users/arman/kaggledata/popcorn'
dfTrain = pd.read_csv(join(root_dir,'labeledTrainData.tsv'),header=0,\
                    delimiter="\t",quoting=3)
dfTest = pd.read_csv(join(root_dir,'testData.tsv'), header=0,\
                   delimiter="\t", quoting=3 )
Explanation: If you are missing bs4 or nltk you can install them via:
pip install bs4
pip install nltk
python -m nltk.downloader all
Setup an I/O directory and put your downloaded data there, we will call this root_dir in the following.
Let's now load the data:
(make sure you change the root_dir to your own path)
End of explanation
dfTrain.head(5)
dfTest.head(5)
Explanation: Let's take a quick look at the data:
End of explanation
dfTrain['review'][11]
Explanation: In particular note that the review column has some html tags:
End of explanation
target = dfTrain['sentiment']
Explanation: Our target is to use sentiment column to predict the same for the test set:
End of explanation
def review_to_wordlist(review, remove_stopwords=False, split=False):
    
    Simple text cleaning function, 
    uses BeautifulSoup to extract text content from html
    removes all non-alphabet
    converts to lower case
    can remove stopwords
    can perform simple tokenization using split by whitespace
    
        
    review_text = BeautifulSoup(review, 'lxml').get_text()
    
    review_text = re.sub("[^a-zA-Z]"," ", review_text)
    
    words = review_text.lower().split()
    
    if remove_stopwords:
        stops = set(stopwords.words("english"))
        words = [w for w in words if not w in stops]
    
    if split:      
        return(words)
    else:
        return(' '.join(words))
Explanation: Now we need some sort of "cleaning" processes, we simply eliminate all the non-alphabet characters and use BeautifulSoup library to extract the text content, Let's put everything together in a function:
End of explanation
review_to_wordlist(dfTrain['review'][11])
Explanation: Before proceeding, let's test what our function does: on the review example above:
End of explanation
review_to_wordlist(dfTrain['review'][11],remove_stopwords=True)
Explanation: and with the remove_stopwords flag on, it will give us:
End of explanation
token = review_to_wordlist(dfTrain['review'][11],remove_stopwords=True, split=True)
print(token)
Explanation: and with split flag on, it can actually perform a simple tokenization:
End of explanation
dfTrain['review'] =  dfTrain['review'].map(review_to_wordlist)
dfTest['review'] =  dfTest['review'].map(review_to_wordlist)
train_len = len(dfTrain)
Explanation: Notice the words
reading, purely, written, 
raised, films, clearly 
that all need stemming, but let's for now continue with what we have
Let's now apply our cleaning process to the review columns:
End of explanation
corpus = list(dfTrain['review']) + list(dfTest['review'])
Explanation: Our corpus is all of the reviews:
End of explanation
tfv = TfidfVectorizer(min_df=3,  max_features=None, ngram_range=(1, 2),\
                      use_idf=True,smooth_idf=True,sublinear_tf=True,\
                      stop_words = 'english')
    
tfv.fit(corpus)
Explanation: Not let's use sklearn's tf-idf vectorizer with unigram and bigrams, and a log TF function (sublinear_tf=True)
Note that we can remove the stop_words here
End of explanation
X_all = tfv.transform(corpus)
Explanation: We can now use the object tfv to build the tf-idf vector-space representation of the reviews, the transformation returns a sparse scipy matrix
Note: Following can take upto 1 min
End of explanation
print(X_all.shape)
Explanation: Notice the shape of the X_all matrix:
End of explanation
train = X_all[:train_len]
test = X_all[train_len:]
Explanation: So it created about 300K numerical features! (the total count of words in the corpus + number of unique bigrams)
It is highly sparse though (which allows python to use scipy's sparse matrix representation and keep everything on the RAM!)
Now let's split the X_all matrix back to our train and test set:
End of explanation
Cs = [1,3,10,30,100,300]
for c in Cs:
    clf = LogisticRegression(penalty='l2', dual=True, tol=0.0001,\
                         C=c, fit_intercept=True, intercept_scaling=1.0,\
                         class_weight=None, random_state=None)
                         
    print("c:",c,"   score:", np.mean(cross_val_score(clf, train, target,\
                            cv=5, scoring='roc_auc')))
Explanation: We now use a Logistic Regression model to fit to the numerical features, (LR is quite safe here to use for such a high number of features, to use tree based models we definitely need feature selection)
Let's perform a simple 5-fold cross-validation using AUC score and also fine tune one of the parameters of the LR model, the penalty constant c
End of explanation
clf = LogisticRegression(penalty='l2', dual=True, tol=0.0001,\
                         C=30, fit_intercept=True, intercept_scaling=1.0,\
                         class_weight=None, random_state=None)
clf.fit(train,target)
Explanation: Our CV experiment suggests that c = 30 is the best choice, so we use our best model to fit to the entire train set now:
End of explanation
preds = clf.predict_proba(test)[:,1]
dfOut = pd.DataFrame( data={"id":dfTest["id"], "sentiment":preds} )
dfOut.to_csv(join(root_dir,'submission.csv'), index=False, quoting=3)
Explanation: and finally predicting for test set and storing the results
End of explanation
<END_TASK>