<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Read in Our Data
The first step is to read out data into a pandas DataFrame.  For an intro to using pandas I would highly suggest looking though this 10 minute guide to pandas.
Step1:  We can now checkout what our data consists of by using the .head() method on our DataFrame.  By default, this will show the top 5 rows.
Step2:  One of the first steps you should take is to get an overview of what kind of data we have but running the .info() method.  Please see the documentation for more info (no pun intended).
Step3:  We can see that the column date_published is being interpreted as an object and not a datetime.  Let's change that by using the pandas.to_datetime() function.
Step4:  Number of Authors
Let's say we wanted to add in another column that contains the number of authors that worked on a particular article.  We could do this like so
Step5:  We can now take a look at the summary statistics of any numeric columns by running the .describe() method.
Step6:  Number of Unique Authors
Let's say we wanted to get the number of unique authors that are represented in this dataframe.  We could potentially use df['author'].nunique() but we are going to run into an error because each row contains a list which isn't hashable.
Instead we could loop through each value and extend a set like so
Step7:  If we also wanted the number of times a particular author was involved in writing an article we could leverage the power of Counter's from the collections library.  Refer to the documentation for more information.
Step8:  Let's say we wanted to now subset down to the articles which Ari Shapiro worked on.  There are a variety of way's we could do this but I will demo one possible avenue.
Step9:  Most popular sections
Let's find what the 5 most popular sections (as judged by the number of articles published within that article)
Step10:  When we first were looking at our DataFrame, you may have noticed that there are quite a few rows missing author information.  Maybe we have a hypothesis that there are certain sections that systemically weren't attaching author information.  Let's dive deeper to try and prove/disprove this hypothesis...
Step11:  As we can see, there are clearly sections that are consistently not attaching author information as well as many that are hit or miss with the author information.
Article Count by Time
Let's make a plot showing the frequency of articles published by day, week, and month.
Step12:  Below we see how we could use the resample function to find the number of articles published per day.  
NOTE
Step13:  There are, of course, many different offset alias' for passing to resample.  For more options see this page.
Step14:  What hour is the most popular time for publishing articles?
To answer this let's extract the hour when the article was published and create a histogram.
Step15:  We were able to run the above command because that particular column contains a datetime object.  From there we can run .dt and then extract any aspect of that datetime (e.g. .dt.hour, .dt.second, .dt.month, .dt.quarter)
Step16:  By default, the .hist method is going to plot 10 bins.  Let's up that to 24 bins so we have a bin for each hour in the day...
Step17:  Selecting Particular Dates
Let's select articles which were published between 10 am and 2 pm on December 24th, 2016.  There are a couple of ways we could do this, but let's start by making a mask.
Step18:  Length of Articles (# Words)
Maybe we are interested in looking at the distribution of how long our articles are...
Step19:  Let's create a histogram of the length of different articles...
Step20:  Clearly there are some outliers in this data.  Let's subset what we are plotting to cut out the top 2% of articles in terms of article length and see what the resulting histogram looks like...
Refer to the numpy percentile function for more information.
Step21:  Only rows that contain 'Obama' in the Headline
We can also use standard string functions by using the .str functionality in pandas.  Take a look at this page for more information.
Step22:  Looking at Average Hour Published by Section
Maybe we have a hypothesis that different sections will vary in the time of day that they are publishing.  We could try and get a sense for this like so
<ASSISTANT_TASK:>
Python Code:
df = pd.read_csv('npr_articles.csv')
Explanation: Read in Our Data
The first step is to read out data into a pandas DataFrame.  For an intro to using pandas I would highly suggest looking though this 10 minute guide to pandas.
End of explanation
df.head()
Explanation: We can now checkout what our data consists of by using the .head() method on our DataFrame.  By default, this will show the top 5 rows.
End of explanation
df.info()
Explanation: One of the first steps you should take is to get an overview of what kind of data we have but running the .info() method.  Please see the documentation for more info (no pun intended).
End of explanation
df['date_published'] = pd.to_datetime(df['date_published'])
df.info()
Explanation: We can see that the column date_published is being interpreted as an object and not a datetime.  Let's change that by using the pandas.to_datetime() function.
End of explanation
# Let's create a mask for all rows that have a non-null value
mask = df['author'].notnull()
# When the data was saved to a csv, these lists were converted into strings, we can convert
# them back like so
from ast import literal_eval
df.loc[mask, 'author'] = df.loc[mask, 'author'].map(literal_eval)
# Initialize column with NaN's and then fill in the respective values
df['num_authors'] = np.nan
df.loc[mask, 'num_authors'] = df.loc[mask, 'author'].map(len)
Explanation: Number of Authors
Let's say we wanted to add in another column that contains the number of authors that worked on a particular article.  We could do this like so:
End of explanation
df.describe()
df.head()
Explanation: We can now take a look at the summary statistics of any numeric columns by running the .describe() method.
End of explanation
# Create a set to hold our authors
authors = set()
for lst in df.loc[mask, 'author']:
    # For every row, update the authors set with those contained in that row
    authors.update(lst)
# Print out the total authors seen
print(len(authors))
Explanation: Number of Unique Authors
Let's say we wanted to get the number of unique authors that are represented in this dataframe.  We could potentially use df['author'].nunique() but we are going to run into an error because each row contains a list which isn't hashable.
Instead we could loop through each value and extend a set like so:
End of explanation
from collections import Counter
authors = df.loc[mask, 'author'].map(Counter).sum()
authors
authors.most_common()
authors['Ari Shapiro']
Explanation: If we also wanted the number of times a particular author was involved in writing an article we could leverage the power of Counter's from the collections library.  Refer to the documentation for more information.
End of explanation
# Because some rows have NaN's in them, we need to get clever with how we
# create our mask
mask = df['author'].map(lambda x: 'Ari Shapiro' in x if isinstance(x, list)
                                   else False)
df.loc[mask, 'headline']
# Here is another way we could acheive this
mask = df.loc[df['author'].notnull(), 'author'].map(lambda x: 'Ari Shapiro' in x)
df.loc[df['author'].notnull()].loc[mask, 'headline']
Explanation: Let's say we wanted to now subset down to the articles which Ari Shapiro worked on.  There are a variety of way's we could do this but I will demo one possible avenue.
End of explanation
df['section'].value_counts(dropna=False)[:5]
Explanation: Most popular sections
Let's find what the 5 most popular sections (as judged by the number of articles published within that article)
End of explanation
# Let's create a new column that indicates whether the author attribute was null or not
# This helps with the groupby below
df['author_null'] = df['author'].isnull()
# Get the mean amount of nulls for each section and sort descending
# NOTE: 1.0 indicates ALL Nulls
df.groupby('section')['author_null'].mean().sort_values(ascending=False)
Explanation: When we first were looking at our DataFrame, you may have noticed that there are quite a few rows missing author information.  Maybe we have a hypothesis that there are certain sections that systemically weren't attaching author information.  Let's dive deeper to try and prove/disprove this hypothesis...
End of explanation
# Create a pandas Series with 1's as the values and the date as the index
s = pd.Series([1], index=df['date_published'])
s[:10]
Explanation: As we can see, there are clearly sections that are consistently not attaching author information as well as many that are hit or miss with the author information.
Article Count by Time
Let's make a plot showing the frequency of articles published by day, week, and month.
End of explanation
# Let's resample that Series and sum the values to find the number of articles by Day
s.resample('D').sum()
Explanation: Below we see how we could use the resample function to find the number of articles published per day.  
NOTE: Our DataFrame/Series must have a datetimeindex for this to work!
End of explanation
plt.plot(s.resample('D').sum())
plt.title('Article Count By Day')
plt.ylabel('Number of Articles')
plt.xlabel('Date')
locs, labels = plt.xticks()
plt.setp(labels, rotation=-45);
plt.plot(s.resample('W').sum())
plt.title('Article Count By Week')
plt.ylabel('Number of Articles')
plt.xlabel('date')
locs, labels = plt.xticks()
plt.setp(labels, rotation=-45);
Explanation: There are, of course, many different offset alias' for passing to resample.  For more options see this page.
End of explanation
df['hour_published'] = df['date_published'].dt.hour
Explanation: What hour is the most popular time for publishing articles?
To answer this let's extract the hour when the article was published and create a histogram.
End of explanation
df['hour_published'].hist()
plt.ylabel('Number of Articles Published')
plt.xlabel('Hour Published (24Hr)');
Explanation: We were able to run the above command because that particular column contains a datetime object.  From there we can run .dt and then extract any aspect of that datetime (e.g. .dt.hour, .dt.second, .dt.month, .dt.quarter)
End of explanation
# Let's force the plot to split into 24 bins, one for each hour
df['hour_published'].hist(bins=24)
plt.ylabel('Number of Articles Published')
plt.xlabel('Hour Published (24Hr)');
# Let's extract the relative frequency rather than the raw counts
df['hour_published'].hist(bins=24, normed=True, alpha=0.75)
plt.ylabel('Freq. of Articles Published')
plt.xlabel('Hour Published (24Hr)');
# We can also grab this information without plotting it using .value_counts
df['hour_published'].value_counts()
df['hour_published'].value_counts(normalize=True)
# Or we could leave them in the order of a day
df['hour_published'].value_counts().sort_index()
Explanation: By default, the .hist method is going to plot 10 bins.  Let's up that to 24 bins so we have a bin for each hour in the day...
End of explanation
mask = ((df['date_published'] >= '2016-12-24 10:00:00') &
        (df['date_published'] <= '2016-12-24 14:00:00'))
df.loc[mask, :]
# Or we could reset or index and do it that way...
df2 = df.set_index('date_published')
df2.loc['2016-12-24 10:00:00': '2016-12-24 14:00:00', :]
Explanation: Selecting Particular Dates
Let's select articles which were published between 10 am and 2 pm on December 24th, 2016.  There are a couple of ways we could do this, but let's start by making a mask.
End of explanation
df['num_words'] = df['article_text'].map(lambda x: len(x.split()))
df['num_words'].describe()
Explanation: Length of Articles (# Words)
Maybe we are interested in looking at the distribution of how long our articles are...
End of explanation
df['num_words'].hist(bins=20, alpha=0.75)
plt.ylabel('Number of Articles Published')
plt.xlabel('Length of Article');
Explanation: Let's create a histogram of the length of different articles...
End of explanation
cutoff = np.percentile(df['num_words'], 98)
df.loc[df['num_words'] <= cutoff, 'num_words'].hist(bins=20, alpha=0.75)
plt.ylabel('Number of Articles Published')
plt.xlabel('Length of Article');
Explanation: Clearly there are some outliers in this data.  Let's subset what we are plotting to cut out the top 2% of articles in terms of article length and see what the resulting histogram looks like...
Refer to the numpy percentile function for more information.
End of explanation
df.loc[df['headline'].str.contains('Obama'), 'headline'].head()
Explanation: Only rows that contain 'Obama' in the Headline
We can also use standard string functions by using the .str functionality in pandas.  Take a look at this page for more information.
End of explanation
# Let's subset to just the 10 most popular sections
top_sections = df['section'].value_counts()[:10].index
df_sub = df.loc[df['section'].isin(top_sections), :]
# We are now grouping by the section and extracting the mean hour that articles were published
df_sub.groupby('section')['hour_published'].mean()
Explanation: Looking at Average Hour Published by Section
Maybe we have a hypothesis that different sections will vary in the time of day that they are publishing.  We could try and get a sense for this like so:
End of explanation
<END_TASK>