<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Sentiment analysis of weather related tweets
This notebook explores some initial modeling of sentiment analysis of weather related tweets. The datasets can be downloaded from Kaggle's "Partly Sunny with a Chance of Hashtags" competition.
Since the competition is over we will restrict ourselves to look at the training set only in order to have a good basis for benchmarking. We will make heavy use of nltk as a natural language processing (NLP) library to build a bag-of-words model that we we will use to train a Multinomial Naive Bayes classifier.
Retrieving and basic cleansing of the data
Step1:  For completeness we will also show a snippet from the test set to see what kind of features we are allowed to use.
Step2:  Label creation for Sentiment category
We start out by trying to model the sentiment category of a tweet. To this end we start by replacing the maximum vlaue of the sentiment fields s1-s5 with the corresponding category
Step3:  Tokenizing, stemming and term counting
In order to use the tweets as input for a machine learning algorithm we need to convert them into numerical features. One way to do this is to chop up the tweets and count the number of words within a tweet and turn them into a large sparse vector whose length is size of the vocabulary of all tweets.
We start with a very simple way of counting the terms in the total of all tweets and count the number of terms.
Step4:  The simple counter already indicates a problem. There are a lot of very common words that have obiously no signal, such as 'the', 'at', 'of', etc. We need to remove those stopwords. Another problem can be seem by comparing '@mention' and '@mention
Step5:  Let's have a look at the word counts again
Step6:  This looks already much better and we can identify weather realated terms such as 'storm', 'sunni', etc.
This will be a good starting point for the rest of the model.
Model generation
One step that is missing is how to use the custom_tokenizer to actually create feature vectors. Luckily enough sklearn provides use with the right functionality to do just that.
Step7:  To ensure we have always the same vectorization options we need to "fit" the vectorizer to the set so we can vectorize any new examples according to this. We do it here on the full set from which we will also extract the cross-validation data and we should be aware of this bias, as it necessarily eliminates the possibility of encountering previously unseen terms when we run our test set predicition.
Step8:  Splitting the set and training the model
As the training data and the labels are now spearate and of different type it is annoying to split the training and test-sets manually. Again sklearn has a tool for us that we can use
Step9:  To avoid the above case of having the "perfect" vocabulary let us now train a new vectorizer using only the training set
Step10:  We can now start to train the model. As this is a highly sparse problem it lends itself to be tackled using a Naive Bayes classifier.
Step11:  With only single terms (monograms), bigrams (interestingly enough they seem not to have any impact at this level) and no additionally fine-grained modeling we already classify 60% of the tweets into the right category. This is impressive in so far that there are a lot of easy and obvious ways to improve the classification. Some ways to improve are
include higher n-grams in the tokenizer
train models per state or per city
include state and city as features
include TF-IDF vectorizers
use a different classifier?
Miscellenea
References
are mostly given throughout the text. But importantly
[1] Partly Sunny with a Chance of Hashtags
[2] scikit-learn
[3] nltk
Stylesheet
<ASSISTANT_TASK:>
Python Code:
import pandas as pd
pd.options.display.encoding = 'ascii'
### import the different corpora and packages from nltk as you need them
# import nltk
# nltk.download()
raw_df = pd.DataFrame.from_csv('./datasets/train.csv')
raw_df.head(5)
Explanation: Sentiment analysis of weather related tweets
This notebook explores some initial modeling of sentiment analysis of weather related tweets. The datasets can be downloaded from Kaggle's "Partly Sunny with a Chance of Hashtags" competition.
Since the competition is over we will restrict ourselves to look at the training set only in order to have a good basis for benchmarking. We will make heavy use of nltk as a natural language processing (NLP) library to build a bag-of-words model that we we will use to train a Multinomial Naive Bayes classifier.
Retrieving and basic cleansing of the data
End of explanation
test_df = pd.DataFrame.from_csv('./datasets/test.csv')
test_df.head(2)
Explanation: For completeness we will also show a snippet from the test set to see what kind of features we are allowed to use.
End of explanation
def create_label(row):
    # row is of time pandas.Series need to cast to a list.
    lst = row.tolist()
    return lst.index(max(lst))+1
# apply defaults to work on columns rather than rows. Use axis = 1 to cahnge to row application.
label_df = raw_df[['s1','s2','s3','s4','s5']].apply(create_label, axis=1)
label_df.head(5)
Explanation: Label creation for Sentiment category
We start out by trying to model the sentiment category of a tweet. To this end we start by replacing the maximum vlaue of the sentiment fields s1-s5 with the corresponding category
End of explanation
from collections import Counter
full_tweet_string = raw_df.tweet.apply(lambda t: t.lower() + " ").sum()
Counter(full_tweet_string.split()).most_common()[:25]
Explanation: Tokenizing, stemming and term counting
In order to use the tweets as input for a machine learning algorithm we need to convert them into numerical features. One way to do this is to chop up the tweets and count the number of words within a tweet and turn them into a large sparse vector whose length is size of the vocabulary of all tweets.
We start with a very simple way of counting the terms in the total of all tweets and count the number of terms.
End of explanation
from nltk.corpus import stopwords
from nltk.stem.snowball import EnglishStemmer
from nltk import word_tokenize
from nltk.util import ngrams
import string
def custom_tokenizer(document_string):
    
    # define the stop word vocabulary
    stops = [unicode(word) for word in stopwords.words('english')] \
        + ["''", "``", 're:', 'fwd:', '-', '@mention', '@mention:', 'mention', 'link', ':', 'f.', '&'] 
    
    # create a default stemmer
    stemmer = EnglishStemmer()
    
    # return the stemmed list of words
    tokens = [stemmer.stem(unicode(word)) for word in word_tokenize(document_string.lower()) \
            if not (unicode(word.lower()) in stops or unicode(word.lower()) in list(string.punctuation))]
    
    return tokens + list(ngrams(tokens, 2))
Explanation: The simple counter already indicates a problem. There are a lot of very common words that have obiously no signal, such as 'the', 'at', 'of', etc. We need to remove those stopwords. Another problem can be seem by comparing '@mention' and '@mention:' which should clearly be identified as the same word, meaning we need to remove the punctuation. Finally we might want to identify 'storm' and 'stormy' as the same and thus require stemming techniques.
The following code snippet provides a tokenizer that does all of the above
End of explanation
Counter(custom_tokenizer(full_tweet_string)).most_common()[:25]
Explanation: Let's have a look at the word counts again
End of explanation
from sklearn.feature_extraction.text import CountVectorizer
vectorizer = CountVectorizer(min_df = 100, max_df = 10000, tokenizer = custom_tokenizer)
Explanation: This looks already much better and we can identify weather realated terms such as 'storm', 'sunni', etc.
This will be a good starting point for the rest of the model.
Model generation
One step that is missing is how to use the custom_tokenizer to actually create feature vectors. Luckily enough sklearn provides use with the right functionality to do just that.
End of explanation
# fit the vectorizer to the full set
X = vectorizer.fit_transform(raw_df.tweet.tolist())
X.shape
Explanation: To ensure we have always the same vectorization options we need to "fit" the vectorizer to the set so we can vectorize any new examples according to this. We do it here on the full set from which we will also extract the cross-validation data and we should be aware of this bias, as it necessarily eliminates the possibility of encountering previously unseen terms when we run our test set predicition.
End of explanation
from sklearn import cross_validation
df_train, df_test, y_train, y_test = cross_validation.train_test_split(raw_df, label_df, test_size = 0.3)
Explanation: Splitting the set and training the model
As the training data and the labels are now spearate and of different type it is annoying to split the training and test-sets manually. Again sklearn has a tool for us that we can use: the cross_validation API.
End of explanation
X_train = vectorizer.fit_transform(pd.DataFrame(df_train)[0].tolist())
X_test = vectorizer.transform(pd.DataFrame(df_test)[0].tolist())
Explanation: To avoid the above case of having the "perfect" vocabulary let us now train a new vectorizer using only the training set
End of explanation
from sklearn.naive_bayes import MultinomialNB
multi_nb_clf = MultinomialNB()
multi_nb_clf.fit(X_train, y_train)
multi_nb_clf.score(X_test, y_test)
Explanation: We can now start to train the model. As this is a highly sparse problem it lends itself to be tackled using a Naive Bayes classifier.
End of explanation
from IPython.core.display import HTML
def css_styling():
    styles = open("../styles/custom.css", "r").read()
    return HTML(styles)
css_styling()
Explanation: With only single terms (monograms), bigrams (interestingly enough they seem not to have any impact at this level) and no additionally fine-grained modeling we already classify 60% of the tweets into the right category. This is impressive in so far that there are a lot of easy and obvious ways to improve the classification. Some ways to improve are
include higher n-grams in the tokenizer
train models per state or per city
include state and city as features
include TF-IDF vectorizers
use a different classifier?
Miscellenea
References
are mostly given throughout the text. But importantly
[1] Partly Sunny with a Chance of Hashtags
[2] scikit-learn
[3] nltk
Stylesheet
End of explanation
<END_TASK>