<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 EDA
Step1:  Trends in Sentiment Scores
Step2:  This plot shows the relationship between title scores and body scores for all articles. Interestingly, there seems to be little relationship between the sentiment score of the title and that of the body. The other feature of note is that article titles have a much wider score range than the bodies. This may be because the compute_score function averages the score of each word over the entire length of the document. Since article bodies contain many more words, and therefore more neutral words, their scores will be lower.
Step3:  This histogram shows the distribution of article bodies' sentiment scores for all three publications. As I suspected, they are normally distributed tightly around zero.
Step4:  Let's see which sections are the most positive and negative. This table contains the average sentiment scores of bodies and titles for each section, sorted by highest-scoring body to lowest. The averages are clustered around zero, but entertainment and education have the most positive content in their article bodies, and articles from the world news setion have the most negative.
Step5:  This histogram shows the distribution of scores for the article bodies from three sections
Step6:  The obvious observation from these graphs is that more opinion pieces mention Trump than not for all three publications, and the sentiment of the Trump-related articles spans a wider range than non-Trump-related articles.
Step7:  Here, I'm comparing opinion pieces from all three publications that mention education to those that don't. Opinion articles about education have a higher rating on average than those that don't, across all publications.
Evaluating Fluctuations in Sentiment Over Time
<ASSISTANT_TASK:>
Python Code:
from articledata import *
data = ArticleData().call()
import pickle
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from datetime import datetime, timedelta
import numpy as np
data = pd.read_pickle('/Users/teresaborcuch/capstone_project/notebooks/pickled_data.pkl')
data.shape
data.head(1)
Explanation: EDA: Sentiment Analysis
By now, I've amassed a corpus of articles from three sources: The New York Times (1,425 articles), The Washington Post (447 articles), and FoxNews.com (309 articles). I'm aiming to have at least 1,000 articles from each source, but I'll start examining trends in the data I currently have.
Sentiment analysis is a natural language processing technique that seeks to identify the polarity of written text - whether the document expresses positive, negative, or neutral feelings. This technique is commonly applied to text that may express strong opinions, such as reviews or social media posts, but I'm going to apply it to news articles and see if there are any differences in the sentiments expressed by different publications or in different sections.
The goal for this analysis is to derive scores for each article's body and title based on the positivity or negativity of the words they contain, and see if these score might be useful features in predicting what section the article is from. Sentiment scores range from 1 (very positive) to -1 (very negative), and I'm expecting a relatively small range for articles in this corpus, since news is presumably objective. SentiWordNet is a resource that maps tens of thousands of English words to a sentiment score. Since language is complex, and the meaning of a given word can vary significantly depending on the context in which it's used, these scores are imperfect, but they'll provide a general picture of the sentiment behind each article.
ArticleData Class
I've been scraping each of my three sources at least twice a day to get fresh news and storing articles in a local postgres database. To facilitate the process of retrieving data and automate some of the cleaning and formatting, I've created a service class ArticleData. Objects of this class have a call method that creates a dataframe of article titles, dates, bodies, sections, and sources. Any articles that have a body of under 200 characters in length are dropped, all dates converted to datetime format, and the various sections from each source are condensed into business, world, politics, science/health, technology, sports, education, entertainment, or opinion. 
Sentiment Scores
Since I'll be looking at sentiment scores, ArticleData objects also calculate scores for article bodies and titles, and store them in columns in the dataframes they return when called. I've borrowed the compute_score function from this Github gist and written it into the call method. This function works by using NLTK's WordNetLemmatizer to lemmatize each word in a text, and the PerceptronTagger to tag the part of speech of each word in the text as a noun, verb, adjective, or adverb. This helps SentiwordNet to assign the word to a synset, or synonym set, for which it can assign a positive or negative value. Since a word can belong to multiple synsets, the function selects the most common one.
End of explanation
plt.scatter(data['SA_title'], data['SA_body'])
plt.xlabel('Sentiment Scores of Article Titles')
plt.ylabel('Sentiment Scores of Article Bodies')
plt.xlim(-0.35, 0.35)
plt.ylim(-0.35, 0.35)
plt.show()
Explanation: Trends in Sentiment Scores
End of explanation
mask1 = (data['source'] == 'Fox')
mask2 = (data['source'] == 'WP')
mask3 = (data['source'] == 'NYT')
score_dict = {'Fox': data[mask1]['SA_body'],
           'WP': data[mask2]['SA_body'],
           'NYT' : data[mask3]['SA_body']}
sns.set_style("whitegrid", {'axes.grid' : False})
fig, ax = plt.subplots(figsize = (10,6))
for x in score_dict.keys():
    ax = sns.distplot(score_dict[x], kde = False, label = x)
    ax.set_xlabel("Distribution of Sentiment Scores by Publication")
ax.legend()
ax.set_xlim(-0.1, 0.1)
plt.show()
Explanation: This plot shows the relationship between title scores and body scores for all articles. Interestingly, there seems to be little relationship between the sentiment score of the title and that of the body. The other feature of note is that article titles have a much wider score range than the bodies. This may be because the compute_score function averages the score of each word over the entire length of the document. Since article bodies contain many more words, and therefore more neutral words, their scores will be lower.
End of explanation
pd.DataFrame(data.pivot_table(
        values = ['SA_title', 'SA_body'], 
        index = 'condensed_section', 
        aggfunc =np.mean)).sort_values('SA_body', ascending = False)
Explanation: This histogram shows the distribution of article bodies' sentiment scores for all three publications. As I suspected, they are normally distributed tightly around zero.
End of explanation
mask1 = (data['condensed_section'] == 'opinion')
mask2 = (data['condensed_section'] == 'entertainment')
mask3 = (data['condensed_section'] == 'world')
score_dict = {'opinion': data[mask1]['SA_body'],
           'entertainment': data[mask2]['SA_body'],
           'world' : data[mask3]['SA_body']}
sns.set_style("whitegrid", {'axes.grid' : False})
fig, ax = plt.subplots(figsize = (10,6))
for x in score_dict.keys():
    ax = sns.distplot(score_dict[x], kde = False, label = x)
    ax.set_xlabel("Distribution of Sentiment Scores by Section")
ax.legend()
ax.set_xlim(-0.1, 0.1)
plt.show()
Explanation: Let's see which sections are the most positive and negative. This table contains the average sentiment scores of bodies and titles for each section, sorted by highest-scoring body to lowest. The averages are clustered around zero, but entertainment and education have the most positive content in their article bodies, and articles from the world news setion have the most negative.
End of explanation
# create dictionaries for each publication
nyt_trump = evaluate_topic(data = data, section = 'opinion', source = 'NYT', topic = 'Trump')
fox_trump = evaluate_topic(data = data, section = 'opinion', source = 'Fox', topic = 'Trump')
wp_trump = evaluate_topic(data = data, section = 'opinion', source = 'WP', topic = 'Trump')
# plot dictionaries
sns.set_style("whitegrid", {'axes.grid' : False})
fig = plt.figure(figsize = (12, 5))
count = 1
label_dict = {1: "NYT", 2: "Fox", 3: "Washington Post"}
for score_dict in [nyt_trump, fox_trump, wp_trump]:
    ax = fig.add_subplot(1, 3, count)
    for key in score_dict.keys():
        ax.hist(score_dict[key], label = key)
        ax.set_xlabel(label_dict[count])
    ax.legend()
    ax.set_xlim(-0.1, 0.1)
    
    count +=1
    
plt.suptitle('Sentiment Scores for Opinion Articles Mentioning "Trump"')
plt.show()
Explanation: This histogram shows the distribution of scores for the article bodies from three sections: opinion, world, and entertainment. The entertainment section has more articles scoring above zero than the other two sections, and the world news section has fewer. This makes sense, since the entertainment section is composed of stores about "lighter" topics such as the arts, dining, or travel, and the world news section might be a bit "heavier". It also seems like the opinion section has a wider range of scores than the other two.
Evaluating Sentiment For Particular Topics
It might be interesting to investigate whether certain publications differ in the sentiment of their reporting on particular topics, or whether the sentiment for a particular topic varies among sections of the same publication. I've written a function that aggregates the sentiment scores for articles from a particular publication or section into a dictionary with two keys: topic and non-topic. These dictionaries are readily plottable, and I can compare among sections and sources.
End of explanation
# Compare sentiment scores for opinion vs non-opinion pieces that mention education
ed_dict = evaluate_topic(data = data, section = 'opinion', topic = 'education')
for key in ed_dict.keys():
    sns.distplot(ed_dict[key], kde = False, label = key)
plt.legend()
plt.show()
print 'Mean Score for Opinion Pieces Mentioning "education": ', np.mean(ed_dict['topic'])
print 'Mean Score for Opinion Pieces Not Mentioning "education": ', np.mean(ed_dict['nontopic'])
Explanation: The obvious observation from these graphs is that more opinion pieces mention Trump than not for all three publications, and the sentiment of the Trump-related articles spans a wider range than non-Trump-related articles.
End of explanation
data.head(1)
class EvaluateTime():
    
    def __init__(self, data = None, section = None, source = None, topic = None, date = None):
        self.data = data
        self.section = section
        self.source = source
        self.topic = topic
        self.date = date
        
    def call(self):
        #self.plot_date_dict, 
        self.range_date_dict = self.make_dict()
        return self
        
    def make_dict(self):
        # define masks
        section_mask = (self.data['condensed_section'] == self.section)
        source_mask = (self.data['source'] == self.source)
        date_mask = (self.data['date'] > self.date)
    
        # initialize lists for plot_date_dict
        topic_scores = []
        dates = []
        
        # initialize other dict
        range_date_dict = {}
    
        if not self.date:
            print "Please select a start date."
        
        # make plot_date_dict from appropriate subset of data
        else:
            if self.section and self.source:
                masked_data = self.data[section_mask & source_mask & date_mask]
                
            elif self.section and (not self.source):
                masked_data = self.data[section_mask & date_mask]
                
            elif self.source and (not self.section):
                masked_data = self.data[source_mask & date_mask]
                
            else:
                masked_data = self.data[date_mask]
                  
            for i, row in masked_data.iterrows():
                
                if self.topic in row[2]:
                    topic_scores.append(row[6])
                    dates.append(row[1])
                    
                    # add to range_date_dict where keys are the dates and the vales are a list of scores
                    if row[1] not in range_date_dict.keys():
                        range_date_dict[row[1]] = [row[6]]
                    elif row[1] in range_date_dict.keys():
                        (range_date_dict[row[1]]).append(row[6])
        #plot_date_dict = {'date': dates, 'score': topic_scores}
        return range_date_dict #plot_date_dict, 
    def plot_time(self):
        
        x = self.range_date_dict.keys()
        x.sort()
        ordered_x = []
        y = []
        for val in x:
            ordered_x.append(val)
            values = self.range_date_dict[val]
            mean = np.mean(values)
            y.append(mean)
        # define upper and lower boundaries for error bars
        upper_bounds = [max(self.range_date_dict[x]) for x in ordered_x]
        lower_bounds = [min(self.range_date_dict[x]) for x in ordered_x]
        
        # define distance for upper error bar
        y_upper = zip(y, upper_bounds)
        upper_error = [abs(pair[0] - pair[1]) for pair in y_upper]
  
        # define distance for lower error bar 
        y_lower = zip(y, lower_bounds)
        lower_error = [abs(pair[0] - pair[1]) for pair in y_lower]
    
        asymmetric_error = [lower_error, upper_error]
        
        plt.plot(ordered_x, y, c = 'r', marker = 'o')
        plt.errorbar(ordered_x, y, yerr = asymmetric_error, ecolor = 'r', capthick = 1)
        plt.xlim(min(ordered_x) + timedelta(days = -1), max(ordered_x) + timedelta(days = 1))
        plt.xticks(rotation = 70)
        plt.show()
et = EvaluateTime(data = data, source = 'Fox', section = 'politics', topic = 'Trump', date = datetime(2017, 1, 24)).call()
et.plot_time()
et = EvaluateTime(data = data, source = 'NYT', section = 'politics', topic = 'Trump', date = datetime(2017, 1, 24)).call()
et.plot_time()
# make it so plot_time returns the graph as an object so that it can be manipulatedimport matplotlib.pyplot as plt
a = [1,3,5,7]
b = [11,-2,4,19]
plt.plot(a,b)
c = [1,3,2,1]
plt.errorbar(a,b,yerr=c, linestyle="None")
plt.show()
Explanation: Here, I'm comparing opinion pieces from all three publications that mention education to those that don't. Opinion articles about education have a higher rating on average than those that don't, across all publications.
Evaluating Fluctuations in Sentiment Over Time
End of explanation
<END_TASK>