<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Sentiment analysis on
H.P. Lovecraft's The Shunned House
For this, we'll use the TextBlob library (http
Step1:  I've already pulled down The Sunned House from Project Gutenberg (https
Step2:  Now we'll go through every sentence in the story and get the 'sentiment' of each one.  Sentiment analysis in TextBlob returns a polarity and a subjectivity number.  Here we'll just extract the polarity
Step3:  Now we instantiate a dataframe by pulling in that csv
Step4:  Let's plot our data! First let's just look at how the sentiment polarity changes from sentence to sentence
Step5:  Very up and down from sentence to sentence!  Some dark sentences (the ones below 0.0 polarity), some positive sentences (greater than 0.0 polarity) but overall kind of hovers around 0.0 polarity.
One thing that may be interesting to look at is how the senitment changes over the course of the book.  To examine that further, I'm going to create a new column in the dataframe which is the cumulative summation of the polarity rating, using the cumsum() pandas method
Step6:  So, now let's plot the results-- How does the sentiment of Lovecraft's story change over the course of the book?
Step7:  The climax of Lovecraft's story appears to be around sentence 255 or so.  Things really drop off at that point and get dark, according to the TextBlob sentiment analysis.
What's the dataframe look like?
Step8:  Let's get some basic statistical information about sentence seniments
Step9:  For fun, let's just see what TextBlob thinks are the most negatively polar sentences in the short story
Step10:  Let's take a quick peak at word frequencies by using the re and collections library.  Here we'll use the Counter() and most_common() methods to return a list of tuples of the most common words in the story
<ASSISTANT_TASK:>
Python Code:
from textblob import TextBlob
import pandas as pd
import pylab as plt
import collections
import re
%matplotlib inline
Explanation: Sentiment analysis on
H.P. Lovecraft's The Shunned House
For this, we'll use the TextBlob library (http://textblob.readthedocs.org/en/dev/) and pandas (http://pandas.pydata.org/)
End of explanation
with open (r'lovecraft.txt', 'r') as myfile:
    shunned = myfile.read()
ushunned = unicode(shunned, 'utf-8')
tb = TextBlob(ushunned)
Explanation: I've already pulled down The Sunned House from Project Gutenberg (https://www.gutenberg.org/wiki/Main_Page) and saved it as a text file called 'lovecraft.txt'.  Here we'll load it then define the encoding as utf-8.  Lastly, we'll instantiate a TextBlob object:
End of explanation
paragraph = tb.sentences
i = -1
for sentence in paragraph:
    i += 1
    pol = sentence.sentiment.polarity
    if i == 0:
        write_type = 'w'
        with open('shunned.csv', write_type) as text_file:
            header = 'number,polarity\n'
            text_file.write(str(header))
    write_type = 'a'
    with open('shunned.csv', write_type) as text_file:
        newline = str(i) + ',' + str(pol) + '\n'
        text_file.write(str(newline))
Explanation: Now we'll go through every sentence in the story and get the 'sentiment' of each one.  Sentiment analysis in TextBlob returns a polarity and a subjectivity number.  Here we'll just extract the polarity:
End of explanation
df = pd.DataFrame.from_csv('shunned.csv')
Explanation: Now we instantiate a dataframe by pulling in that csv:
End of explanation
df.polarity.plot(figsize=(12,5), color='b', title='Sentiment Polarity for HP Lovecraft\'s The Shunned House')
plt.xlabel('Sentence number')
plt.ylabel('Sentiment polarity')
Explanation: Let's plot our data! First let's just look at how the sentiment polarity changes from sentence to sentence:
End of explanation
df['cum_sum'] = df.polarity.cumsum()
Explanation: Very up and down from sentence to sentence!  Some dark sentences (the ones below 0.0 polarity), some positive sentences (greater than 0.0 polarity) but overall kind of hovers around 0.0 polarity.
One thing that may be interesting to look at is how the senitment changes over the course of the book.  To examine that further, I'm going to create a new column in the dataframe which is the cumulative summation of the polarity rating, using the cumsum() pandas method:
End of explanation
df.cum_sum.plot(figsize=(12,5), color='r', 
                title='Sentiment Polarity cumulative summation for HP Lovecraft\'s The Shunned House')
plt.xlabel('Sentence number')
plt.ylabel('Cumulative sum of sentiment polarity')
Explanation: So, now let's plot the results-- How does the sentiment of Lovecraft's story change over the course of the book?
End of explanation
df.head()
Explanation: The climax of Lovecraft's story appears to be around sentence 255 or so.  Things really drop off at that point and get dark, according to the TextBlob sentiment analysis.
What's the dataframe look like?
End of explanation
df.describe()
Explanation: Let's get some basic statistical information about sentence seniments:
End of explanation
for i in df[df.polarity < -0.5].index:
    print i, tb.sentences[i]
words = re.findall(r'\w+', open('lovecraft.txt').read().lower())
collections.Counter(words).most_common(10)
Explanation: For fun, let's just see what TextBlob thinks are the most negatively polar sentences in the short story:
End of explanation
words = re.findall(r'\w+', ushunned.lower())
common = collections.Counter(words).most_common()
df_freq = pd.DataFrame(common, columns=['word', 'freq'])
df_freq.set_index('word').head()
Explanation: Let's take a quick peak at word frequencies by using the re and collections library.  Here we'll use the Counter() and most_common() methods to return a list of tuples of the most common words in the story:
End of explanation
<END_TASK>