<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 And the test data.
Step1:  Let's count how many labels do we have for each sentiment class.
Step2:  Finally, let's calculate the average number of words per sentence. We could do the following using a list comprehension with the number of words per sentence.
Step3:  First we need to init the vectorizer. We need to remove puntuations, lowercase, remove stop words, and stem words. All these steps can be directly performed by CountVectorizer if we pass the right parameter values. We can do as follows.
Step4:  Numpy arrays are easy to work with, so convert the result to an array.
Step5:  We can also print the counts of each word in the vocabulary as follows.
Step6:  A bag-of-words linear classifier
Step7:  Now we are ready to train our classifier.
Step8:  Now we use the classifier to label our evaluation set. We can use either predict for classes or predict_proba for probabilities.
Step9:  Finally, we can re-train our model with all the training data and use it for sentiment classification with the original (unlabeled) test set.
<ASSISTANT_TASK:>
Python Code:
test_data_df.head()
Explanation: And the test data.
End of explanation
train_data_df.Sentiment.value_counts()
Explanation: Let's count how many labels do we have for each sentiment class.
End of explanation
import numpy as np 
np.mean([len(s.split(" ")) for s in train_data_df.Text])
Explanation: Finally, let's calculate the average number of words per sentence. We could do the following using a list comprehension with the number of words per sentence.
End of explanation
import re, nltk
from sklearn.feature_extraction.text import CountVectorizer        
from nltk.stem.porter import PorterStemmer
stemmer = PorterStemmer()
def stem_tokens(tokens, stemmer):
    stemmed = []
    for item in tokens:
        stemmed.append(stemmer.stem(item))
    return stemmed
def tokenize(text):
    # remove non letters
    text = re.sub("[^a-zA-Z]", " ", text)
    # tokenize
    tokens = nltk.word_tokenize(text)
    # stem
    stems = stem_tokens(tokens, stemmer)
    return stems
######## 
vectorizer = CountVectorizer(
    analyzer = 'word',
    tokenizer = tokenize,
    lowercase = True,
    stop_words = 'english',
    max_features = 100
)
corpus_data_features = vectorizer.fit_transform(train_data_df.Text.tolist() + test_data_df.Text.tolist())
Explanation: First we need to init the vectorizer. We need to remove puntuations, lowercase, remove stop words, and stem words. All these steps can be directly performed by CountVectorizer if we pass the right parameter values. We can do as follows.
End of explanation
corpus_data_features_nd = corpus_data_features.toarray()
corpus_data_features_nd.shape
vocab = vectorizer.get_feature_names()
print(vocab)
Explanation: Numpy arrays are easy to work with, so convert the result to an array.
End of explanation
dist = np.sum(corpus_data_features_nd, axis=0)
for tag, count in zip(vocab, dist):
    print(count, tag)
Explanation: We can also print the counts of each word in the vocabulary as follows.
End of explanation
from sklearn.cross_validation import train_test_split
# remember that corpus_data_features_nd contains all of our original train and test data, so we need to exclude
# the unlabeled test entries
X_train, X_test, y_train, y_test  = train_test_split(
    corpus_data_features_nd[0:len(train_data_df)], 
    train_data_df.Sentiment,
    train_size=0.85, 
    random_state=1234)
print(X_test[0])
Explanation: A bag-of-words linear classifier
End of explanation
from sklearn.linear_model import LogisticRegression
log_model = LogisticRegression()
log_model = log_model.fit(X=X_train, y=y_train)
Explanation: Now we are ready to train our classifier.
End of explanation
y_pred = log_model.predict(X_test)
from sklearn.metrics import classification_report
print(classification_report(y_test, y_pred))
Explanation: Now we use the classifier to label our evaluation set. We can use either predict for classes or predict_proba for probabilities.
End of explanation
log_model = LogisticRegression()
log_model = log_model.fit(X=corpus_data_features_nd[0:len(train_data_df)], y=train_data_df.Sentiment)
test_pred = log_model.predict(corpus_data_features_nd[len(train_data_df):])
actual_pred = test_data_df["Sentiment"].tolist()
print(classification_report(test_pred, actual_pred))
Explanation: Finally, we can re-train our model with all the training data and use it for sentiment classification with the original (unlabeled) test set.
End of explanation
<END_TASK>