<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Predicting sentiment from product reviews
Fire up GraphLab Create
Step1:  Read some product review data
Loading reviews for a set of baby products.
Step2:  Let's explore this data together
Data includes the product name, the review text and the rating of the review.
Step3:  Build the word count vector for each review
Step4:  Examining the reviews for most-sold product
Step5:  Build a sentiment classifier
Step6:  Define what's a positive and a negative sentiment
We will ignore all reviews with rating = 3, since they tend to have a neutral sentiment.  Reviews with a rating of 4 or higher will be considered positive, while the ones with rating of 2 or lower will have a negative sentiment.
Step7:  Let's train the sentiment classifier
Step8:  Evaluate the sentiment model
Step9:  Applying the learned model to understand sentiment for Giraffe
Step10:  Sort the reviews based on the predicted sentiment and explore
Step11:  Most positive reviews for the giraffe
Step12:  Show most negative reviews for giraffe
Step13:  Exercise
1. Use .apply() to build a new feature with the counts for each of the selected_words
Step14:  Using the .sum() method on each of the new columns you created, answer the following questions
Step15:  2. Create a new sentiment analysis model using only the selected_words as features
Step16:  You will now examine the weights the learned classifier assigned to each of the 11 words in selected_words and gain intuition as to what the ML algorithm did for your data using these features. In GraphLab Create, a learned model, such as the selected_words_model, has a field 'coefficients', which lets you look at the learned coefficients. You can access it by using
Step17:  Using this approach, sort the learned coefficients according to the ‘value’ column using .sort(). Out of the 11 words in selected_words, which one got the most positive weight? Which one got the most negative weight? Do these values make sense for you? Save these results to answer the quiz at the end.
Step18:  3. Comparing the accuracy of different sentiment analysis model
Step19:  4. Interpreting the difference in performance between the models
<ASSISTANT_TASK:>
Python Code:
import graphlab
Explanation: Predicting sentiment from product reviews
Fire up GraphLab Create
End of explanation
products = graphlab.SFrame('amazon_baby.gl/')
Explanation: Read some product review data
Loading reviews for a set of baby products.
End of explanation
products.head()
Explanation: Let's explore this data together
Data includes the product name, the review text and the rating of the review.
End of explanation
products['word_count'] = graphlab.text_analytics.count_words(products['review'])
products.head()
graphlab.canvas.set_target('ipynb')
products['name'].show()
Explanation: Build the word count vector for each review
End of explanation
giraffe_reviews = products[products['name'] == 'Vulli Sophie the Giraffe Teether']
len(giraffe_reviews)
giraffe_reviews['rating'].show(view='Categorical')
Explanation: Examining the reviews for most-sold product:  'Vulli Sophie the Giraffe Teether'
End of explanation
products['rating'].show(view='Categorical')
Explanation: Build a sentiment classifier
End of explanation
#ignore all 3* reviews
products = products[products['rating'] != 3]
#positive sentiment = 4* or 5* reviews
products['sentiment'] = products['rating'] >=4
products.head()
Explanation: Define what's a positive and a negative sentiment
We will ignore all reviews with rating = 3, since they tend to have a neutral sentiment.  Reviews with a rating of 4 or higher will be considered positive, while the ones with rating of 2 or lower will have a negative sentiment.
End of explanation
train_data,test_data = products.random_split(.8, seed=0)
sentiment_model = graphlab.logistic_classifier.create(train_data,
                                                     target='sentiment',
                                                     features=['word_count'],
                                                     validation_set=test_data)
Explanation: Let's train the sentiment classifier
End of explanation
sentiment_model.evaluate(test_data)
sentiment_model.evaluate(test_data, metric='roc_curve')
sentiment_model.show(view='Evaluation')
Explanation: Evaluate the sentiment model
End of explanation
giraffe_reviews['predicted_sentiment'] = sentiment_model.predict(giraffe_reviews, output_type='probability')
giraffe_reviews.head()
Explanation: Applying the learned model to understand sentiment for Giraffe
End of explanation
giraffe_reviews = giraffe_reviews.sort('predicted_sentiment', ascending=False)
giraffe_reviews.head()
Explanation: Sort the reviews based on the predicted sentiment and explore
End of explanation
giraffe_reviews[0]['review']
giraffe_reviews[1]['review']
Explanation: Most positive reviews for the giraffe
End of explanation
giraffe_reviews[-1]['review']
giraffe_reviews[-2]['review']
Explanation: Show most negative reviews for giraffe
End of explanation
selected_words = ['awesome', 'great', 'fantastic', 'amazing', 'love', 'horrible', 'bad', 'terrible', 'awful', 'wow', 'hate']
def awesome_count(word_count):
    if 'awesome' in word_count:
        return word_count['awesome']
    return 0
products['awesome'] = products['word_count'].apply(awesome_count)
def great_count(word_count):
    if 'great' in word_count:
        return word_count['great']
    return 0
products['great'] = products['word_count'].apply(great_count)
def fantastic_count(word_count):
    if 'fantastic' in word_count:
        return word_count['fantastic']
    return 0
products['fantastic'] = products['word_count'].apply(fantastic_count)
def amazing_count(word_count):
    if 'amazing' in word_count:
        return word_count['amazing']
    return 0
products['amazing'] = products['word_count'].apply(amazing_count)
def love_count(word_count):
    if 'love' in word_count:
        return word_count['love']
    return 0
products['love'] = products['word_count'].apply(love_count)
def horrible_count(word_count):
    if 'horrible' in word_count:
        return word_count['horrible']
    return 0
products['horrible'] = products['word_count'].apply(horrible_count)
def bad_count(word_count):
    if 'bad' in word_count:
        return word_count['bad']
    return 0
products['bad'] = products['word_count'].apply(bad_count)
def terrible_count(word_count):
    if 'terrible' in word_count:
        return word_count['terrible']
    return 0
products['terrible'] = products['word_count'].apply(terrible_count)
def awful_count(word_count):
    if 'awful' in word_count:
        return word_count['awful']
    return 0
products['awful'] = products['word_count'].apply(awful_count)
def wow_count(word_count):
    if 'wow' in word_count:
        return word_count['wow']
    return 0
products['wow'] = products['word_count'].apply(wow_count)
def hate_count(word_count):
    if 'hate' in word_count:
        return word_count['hate']
    return 0
products['hate'] = products['word_count'].apply(hate_count)
# products['awesome'] = products['word_count'].apply(awesome_count)
# # Generalize function for apply
# def selected_words_count(word_count, word):
#     if word in word_count:
#         return word_count[word]
#     return 0    
# for word in selected_words:
#     products[word] = products.apply(lambda x: selected_words_count(x['word_count'], word))
products.head()
Explanation: Exercise
1. Use .apply() to build a new feature with the counts for each of the selected_words:
In the notebook above, we created a column ‘word_count’ with the word counts for each review. Our first task is to create a new column in the products SFrame with the counts for each selected_word above, and, in the process, we will see how the method .apply() can be used to create new columns in our data (our features) and how to use a Python function, which is an extremely useful concept to grasp!
End of explanation
print 'Word count value:'
for word in selected_words:
    print '{0}: {1}'.format(word, products[word].sum())
    
   
# awesome: 2002
# great: 42420.0
# fantastic: 873
# amazing: 1305
# love: 40277.0
# horrible: 659
# bad: 3197
# terrible: 673
# awful: 345
# wow: 131
# hate: 1057
Explanation: Using the .sum() method on each of the new columns you created, answer the following questions: Out of the selected_words, which one is most used in the dataset? Which one is least used? Save these results to answer the quiz at the end.
End of explanation
train_data,test_data = products.random_split(.8, seed=0)
selected_words_model = graphlab.logistic_classifier.create(train_data,
                                                     target='sentiment',
                                                     features=selected_words,
                                                     validation_set=test_data)
Explanation: 2. Create a new sentiment analysis model using only the selected_words as features:
In the IPython Notebook above, we used word counts for all words as features for our sentiment classifier. Now, you are just going to use the selected_words:
End of explanation
coef = selected_words_model['coefficients']
Explanation: You will now examine the weights the learned classifier assigned to each of the 11 words in selected_words and gain intuition as to what the ML algorithm did for your data using these features. In GraphLab Create, a learned model, such as the selected_words_model, has a field 'coefficients', which lets you look at the learned coefficients. You can access it by using:
End of explanation
coef = coef.sort('value', ascending=False)
coef
coef.sort('value', ascending=True)
Explanation: Using this approach, sort the learned coefficients according to the ‘value’ column using .sort(). Out of the 11 words in selected_words, which one got the most positive weight? Which one got the most negative weight? Do these values make sense for you? Save these results to answer the quiz at the end.
End of explanation
selected_words_model.evaluate(test_data)
sentiment_model.evaluate(test_data)
selected_words_model.evaluate(test_data, metric='roc_curve')
selected_words_model.show(view='Evaluation')
Explanation: 3. Comparing the accuracy of different sentiment analysis model:
What is the accuracy of the selected_words_model on the test_data? What was the accuracy of the sentiment_model that we learned using all the word counts in the IPython Notebook above from the lectures? What is the accuracy majority class classifier on this task? How do you compare the different learned models with the baseline approach where we are just predicting the majority class? Save these results to answer the quiz at the end.
Hint: we discussed the majority class classifier in lecture, which simply predicts that every data point is from the most common class. This is baseline is something we definitely want to beat with models we learn from data.
End of explanation
diaper_champ_reviews = products[products['name'] == 'Baby Trend Diaper Champ']
diaper_champ_reviews.head()
diaper_champ_reviews['predicted_sentiment'] = sentiment_model.predict(diaper_champ_reviews, output_type='probability')
diaper_champ_reviews = diaper_champ_reviews.sort('predicted_sentiment', ascending=False)
diaper_champ_reviews.head()
diaper_champ_reviews['predicted_sentiment'].max()
selected_words_model.predict(diaper_champ_reviews[0:1], output_type='probability')
# diaper_champ_reviews['predicted_sentiment_2']  = selected_words_model.predict(diaper_champ_reviews, output_type='probability')
diaper_champ_reviews.head()
diaper_champ_reviews[0]['review']
diaper_champ_reviews[0]['word_count']
diaper_champ_reviews[1]['review']
diaper_champ_reviews[-1]['review']
Explanation: 4. Interpreting the difference in performance between the models:
To understand why the model with all word counts performs better than the one with only the selected_words, we will now examine the reviews for a particular product.
We will investigate a product named ‘Baby Trend Diaper Champ’. (This is a trash can for soiled baby diapers, which keeps the smell contained.)
Just like we did for the reviews for the giraffe toy in the IPython Notebook in the lecture video, before we start our analysis you should select all reviews where the product name is ‘Baby Trend Diaper Champ’. Let’s call this table diaper_champ_reviews.
Again, just as in the video, use the sentiment_model to predict the sentiment of each review in diaper_champ_reviews and sort the results according to their ‘predicted_sentiment’.
What is the ‘predicted_sentiment’ for the most positive review for ‘Baby Trend Diaper Champ’ according to the sentiment_model from the IPython Notebook from lecture? Save this result to answer the quiz at the end.
Now use the selected_words_model you learned using just the selected_words to predict the sentiment most positive review you found above. Hint: if you sorted the diaper_champ_reviews in descending order (from most positive to most negative), this command will be helpful to make the prediction you need:
End of explanation
<END_TASK>