<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Week 3 Review
basic numpy manipulation
Write two ways to get the diagonal elements of a dot product of two matrices, A and B.
Step1:  Given some matrix (row normalized to 1.0)
Step2:  Week 4 - Strings
<ul>
<li>Text analysis is a *huge* part of data analysis - its not just numbers
<li>Need to standardize variables, resolve ambiguous labels
<li>Converting text to numeric representations (word bagging, n-grams) is common
</ul>
Today
Tweet example
string modules
<ol>
<li>os
<li>collections
</ol>
Regular Expressions
Review Problems
And if there is time - either more Regression or intro Pandas
Sentiment analysis on Obama's tweets<a id="tweet"></a>
<ul>
<li>One of Python’s strengths is the ease of working with text
</ul>
Step3:  How can we get a handle on all this text?
String methods
<ul>
<li>`Split` a string according to some character. This creates a list of strings out of the original string.
<li>`Join` a list of strings together according to some delimiter     
<li>`Strip` leading and trailing blanks around a word  
<li>`count` the number of exact matches
<li>`find` the index of first occurrence. `-1` if doesn't exist
<li>`replace` replaces every occurrence of one pattern with another
</ul>
Step5:  Basic string manipulation - example
Pull out the methods from the markdown text below
Step6:  +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
EXERCISE TIME!
Format your PATH variable such that it looks like this
Step7:  +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Assess positive or negative sentiment
Step8:  What's the positive sentiment in the tweets?
<ul>
<li>Iterate through tweets and count occurrence of positive words. 
<li>Need to do a bit a cleaning first.
</ul>
Step9:  os module
<ul>
<li>`os` module provides a way of using operating system dependent functionality. You can find important information about your location or about the process.
<li>We'll look at this in more detail next session
</ul>
Step10:  collections modules
<ul>
<li>Very useful module for text manipulation, analysis. Lots of iterating over dictionary-like objects.
<li>`OrderedDict`
<li>`defaultdict`
<li>`namedtuple`
<li>`Counter`
</ul>
Step11:  Digression on Sets
<ul>
<li>Another python data type
<li>Can use `{some_sequence}` or `set(some_list)` to create it
<li>Like lists or tuples, but returns **unique** elements. This can be super handy.
<li>Has a specific way of thinking about it
<li>`add`, `intersection`, `union`
</ul>
Step12:  How to find 10 most common words?
Step13:  To find top 10, need to sort dictionary on values. There are <em>MANY</em> ways to do this.
Step14:  ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
EXERCISE TIME!
Find the number of words that appear exactly 10 times in the book. Use a python module.
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Step15:  Find positive sentiment from tweets
<ul>
<li>Helpful to standardize tweets
<li>Convert to lower case
<li>Remove punctuation
</ul>
Step16:  Find the most common positive word used in the tweets
Step17:  ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
EXERCISE TIME!
Do a similar analysis above but using negative words. 
1) Read in the negative.txt file and store it in a variable call it negative.<br>
2) Using negative, find the 10 most common negative words in the tweets. Comment on any interesting observations.<br>
3) For the most common negative word, print 10 tweets that contain it. Comment on any interesting observations.
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Step18:  What happened??
Regular Expression <a id="regex"></a>
<ul>
<li>Special sequence of characters that helps you match (**exact!**) or find other strings or sets of strings, using a specialized syntax held in a pattern
<li>Regular expressions as a concept is not exclusive to Python at all.
<li>Regular expressions are widely used in UNIX world.
<li>`re` module
        <ul>
        <li>`re.compile()`
        <li>`re.match()`
        <li>`re.search()`
        <li>`re.findall()`
        <li>`re.sub()`
        </ul>
<li>In python, regular expressions tend to be **very slow**. So only use if necessary.
</ul>
Step19:  match - only find matches if they occur at the start of the string being searched.
Step20:  search - similar to match, but search doesn’t restrict us to only finding matches at the beginning of the string. But will only return the first result
Step21:  findall - get a list of all matching patterns
Step22:  Use compile to create regular expressions that you can then search, findall, match, ...
Step23:  ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
EXERCISE TIME!
Redo the negative tweet exercise from above to perform exact matching. That is
Step24:  What is the distribution of negative word occurrence?
Let's plot it!
Step25:  Is this chart helpful?
Where are the words??
Step26:  Distribution of word length
Step27:  Anyone notice something strange about the chart?
Step28:  n-grams
<ul>
<li>Single word counts is ok, but for richer text analysis might want combinations of words
<li>bi-grams, tri-grams, ..., n-grams
</ul>
Step29:  +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
EXERCISE TIME
Write your own bigram function. Do this without using a for loop. Hint - look up the zip function.
Do the same for tri grams.
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Step30:  Take Away
<ul>
<li>Manipulating strings and text analysis
<li>Reading files from the web and local 
<li>Basic plotting
</ul>
Review Problems<a id="hmwk"></a>
Q1.
Write a function to remove vowels from a string. Stick to base python only.
Step31:  Q2.
For the next few questions, read in Through the Looking glass from Project Gutenberg (pg12.txt).
Step32:  A). Print the subset starting from 'JABBERWOCKY' and ending with 'It seems very pretty.'
Step33:  B). Print all distinct palindromic words, if any. Remember to 'standardize' first.
Step35:  C). Find all possible sequences of 3 words in the poem. This is called a trigram. (Hint
Step36:  Q3.
Plot a similar distribution for the positive word count. The differences are
Step37:  BONUS QUESTION!
Q1. Encode and decode Through the Looking Glass using a Caesar cipher <br>
Hint
<ASSISTANT_TASK:>
Python Code:
import numpy as np
A = np.arange(25).reshape(5,5)
B = np.arange(25).reshape(5,5)
A
print np.diag(np.dot(A, B))
B.T
A * B.T
print np.sum(A * B.T, axis=1)  # what does this do?
Explanation: Week 3 Review
basic numpy manipulation
Write two ways to get the diagonal elements of a dot product of two matrices, A and B.
End of explanation
P = np.array([[ 0.30434783, 0.34782609, 0.34782609],[0.08333333,  0.25, 0.66666667], [ 0.75, 0.16666667,  0.08333333]])
2^5
2*2*2*2*2
def raise_nth_power(n, transition):
    ''' Solve for stationary distribution by raising to the nth power '''
    
    temp = transition
    for i in np.arange(n-1):    
        temp = transition.dot(temp)
    
    return temp
n = 50
print raise_nth_power(n,P)
# how to run a linear regression using scikit learn. See week 3 notebook for example.
Explanation: Given some matrix (row normalized to 1.0):
python
 P = [[ 0.30434783  0.34782609  0.34782609]
     [ 0.08333333  0.25        0.66666667]
     [ 0.75        0.16666667  0.08333333]]
Find the stationary distribution.
End of explanation
import urllib     # popular module for fetching data from web
files = ['positive.txt','obama_tweets.txt']     # create list of filenames
path = 'http://www.unc.edu/~ncaren/haphazard/'   # this URL has good text documents to practice on
for file_name in files:
     urllib.urlretrieve(path + file_name,file_name)    # pull down text data from URL
tweet_file = open("obama_tweets.txt").read()
tweet_file[:1000]
Explanation: Week 4 - Strings
<ul>
<li>Text analysis is a *huge* part of data analysis - its not just numbers
<li>Need to standardize variables, resolve ambiguous labels
<li>Converting text to numeric representations (word bagging, n-grams) is common
</ul>
Today
Tweet example
string modules
<ol>
<li>os
<li>collections
</ol>
Regular Expressions
Review Problems
And if there is time - either more Regression or intro Pandas
Sentiment analysis on Obama's tweets<a id="tweet"></a>
<ul>
<li>One of Python’s strengths is the ease of working with text
</ul>
End of explanation
tweets = tweet_file.split('\n')   # very commonly used
tweets[:10]
print len(tweets)
tweets[:5]
from __future__ import division
# Average number of hashtags total?
# I hope you remember your list comprehensions :)
sum([tweet.count('#') for tweet in tweets])/len(tweets)      # exact matching pattern
print tweets[300]
print tweets[300].find('a')  # only first occurrence
print tweets[300].find('!')  # -1 if doesn't occur
# Number of tweets with no #?
sum([tweet.find('#') == -1  for tweet in tweets])       
tweets[300].replace('Obama', "Trump")
Explanation: How can we get a handle on all this text?
String methods
<ul>
<li>`Split` a string according to some character. This creates a list of strings out of the original string.
<li>`Join` a list of strings together according to some delimiter     
<li>`Strip` leading and trailing blanks around a word  
<li>`count` the number of exact matches
<li>`find` the index of first occurrence. `-1` if doesn't exist
<li>`replace` replaces every occurrence of one pattern with another
</ul>
End of explanation
objective = "Basic string manipulation"
s = 
<ul>
  <li>`Split` a string according to some character. This creates a list of strings out of the original string.
  <li>`Join` a list of strings together according to some delimiter     
  <li>`Strip` leading and trailing blanks around a word  
  <li>`count` the number of exact matches
  <li>`find` the index of first occurrence. `-1` if doesn't exist
  <li>`replace` replaces every occurrence of one pattern with another
</ul>
s     # where's the newline characters?
split_list = [i for i in s.split('\n')]
split_list
split_list[2:-2]
strip_list = [i.strip('   <li>') for i in split_list[2:-2]] # remove trailing space and markdown characters
strip_list
# should remove first two and last two
tokenize = [i.split(' ') for i in strip_list]
print tokenize
final_words = [i[0].strip('`') for i in tokenize]
final_words
objective
" + ".join(final_words) + " = {}".format(objective)
Explanation: Basic string manipulation - example
Pull out the methods from the markdown text below:
objective = "Basic string manipulation"
<ul>
  <li>`Split` a string according to some character. This creates a list of strings out of the original string.
  <li>`Join` a list of strings together according to some delimiter     
  <li>`Strip` leading and trailing blanks around a word  
  <li>`count` the number of exact matches
  <li>`find` the index of first occurrence. `-1` if doesn't exist
  <li>`replace` replaces every occurrence of one pattern with another
</ul>
End of explanation
paths = !`echo $PATH`
print paths     # paths is a list of strings
print len(paths)
path_strings = paths[0]
print path_strings
all_paths = path_strings.split(':')  # split on : which is what separates paths
print all_paths
for path in all_paths:
    print '=> '.join(path.strip().split('/'))
Explanation: +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
EXERCISE TIME!
Format your PATH variable such that it looks like this:
=&gt; bin=&gt; sh
=&gt; Users=&gt; ilanman=&gt; sparkapphelper=&gt; scala-2.11.7=&gt; bin=&gt; bin
=&gt; Users=&gt; ilanman=&gt; sparkapphelper=&gt; spark=&gt; spark-1.6.0-bin-hadoop2.6=&gt; bin
=&gt; Users=&gt; ilanman=&gt; sparkapphelper=&gt; sbt=&gt; sbt-0.13.11=&gt; bin
=&gt; Library=&gt; Java=&gt; JavaVirtualMachines=&gt; jdk1.8.0_60.jdk=&gt; Contents=&gt; Home=&gt; bin
=&gt; usr=&gt; local=&gt; bin
=&gt; usr=&gt; bin
=&gt; bin
=&gt; usr=&gt; sbin
=&gt; sbin
=&gt; opt=&gt; X11=&gt; bin
=&gt; Library=&gt; TeX=&gt; texbin
No such file or directory
Note that Jupyter can access your terminal by entering ! before regular terminal commands, i.e. !mkdir newfolder<br>
Here is my current PATH: !echo $PATH
/Users/ilanman/spark/bin:/Users/ilanman/sbt/bin:/Library/Java/JavaVirtualMachines/jdk1.8.0_60.jdk/Contents/Home/bin<br>
/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/Library/TeX/texbin
End of explanation
positive_file = open("positive.txt").read()
positive_file[:1000]
positive_file.split('\n')[:10]
Explanation: +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Assess positive or negative sentiment
End of explanation
tweets[0]
# The punctuation function is useful for getting rid of unwanted characters in a string
from string import punctuation
print punctuation
Explanation: What's the positive sentiment in the tweets?
<ul>
<li>Iterate through tweets and count occurrence of positive words. 
<li>Need to do a bit a cleaning first.
</ul>
End of explanation
import os
print os.getcwd()
# Alice in Wonderland from Project Gutenberg
if not os.path.exists('alice.txt'):
    ! curl http://www.gutenberg.org/cache/epub/11/pg11.txt -O alice.txt
!head -50 pg11.txt
Explanation: os module
<ul>
<li>`os` module provides a way of using operating system dependent functionality. You can find important information about your location or about the process.
<li>We'll look at this in more detail next session
</ul>
End of explanation
alice = open('pg11.txt').read()  # read in file
# split words on spaces, remove punctuation and make lower case
words = alice.translate(None, punctuation).lower().split()    # common to chain together string methods
print len(words)
print len(set(words))   # set - unique elements
Explanation: collections modules
<ul>
<li>Very useful module for text manipulation, analysis. Lots of iterating over dictionary-like objects.
<li>`OrderedDict`
<li>`defaultdict`
<li>`namedtuple`
<li>`Counter`
</ul>
End of explanation
x = [1,2,3,2,1,2,3,4,5,6,7,6,5,4,3,3]  # some list
set(x)
y = {1,2,3,2,1,2,3,4,5,6,7,6,5,4,3,3}
y
set(['this is a sentence'])
set('this is a sentence')
a = [1,2,3,4,5]
b = [3,4,5,6,7]
set(a).intersection(set(b))
set(a).union(set(b))
set(b).difference(set(a))   # a - b
Explanation: Digression on Sets
<ul>
<li>Another python data type
<li>Can use `{some_sequence}` or `set(some_list)` to create it
<li>Like lists or tuples, but returns **unique** elements. This can be super handy.
<li>Has a specific way of thinking about it
<li>`add`, `intersection`, `union`
</ul>
End of explanation
most_common = {}
# use set to find unique words, store in dictionary
for word in set(words):
    most_common.update({word : words.count(word)})     # WHAT DOES UPDATE DO?
most_common
Explanation: How to find 10 most common words?
End of explanation
most_common.items()
# using base python
sorted_words_lambda = sorted(most_common.items(), key = lambda x: x[1], reverse = True)
sorted_words_lambda[:10]
# using itemgetter from operator module
from operator import itemgetter
sorted_words_itemgetter = sorted(most_common.items(), key = itemgetter(1), reverse = True)
sorted_words_itemgetter[:10]
# using collections module
from collections import OrderedDict
sorted_words_collections = OrderedDict(sorted(most_common.items(), key = lambda x: x[1], reverse = True))
sorted_words_collections.items()[:10]
# default dict
from collections import defaultdict
inverse = defaultdict( list )
for k, v in most_common.items():
    inverse[v].append( k )
for k in sorted(inverse,reverse=True)[:10]:
    print inverse[k][0], k
# named tuple
from collections import namedtuple
sorted_words_named = namedtuple('word', 'word count')
sorted_words_named = sorted([sorted_words_named(v,k) for (k,v) in most_common.items()], reverse=True)
sorted_words_named[:10]
# using Counter
from collections import Counter
sorted_words_counter = [item for item in Counter(words).most_common(10)]
sorted_words_counter
Explanation: To find top 10, need to sort dictionary on values. There are <em>MANY</em> ways to do this.
End of explanation
## SOLUTION
alice_counter = Counter(alice.split())
len([(k, v) for (k, v) in alice_counter.items() if v==10])
Explanation: ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
EXERCISE TIME!
Find the number of words that appear exactly 10 times in the book. Use a python module.
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
End of explanation
positive_word_dict = {}
positives = positive_file.split('\n')[::-1][1:]
for tweet in tweets:   # for all tweets
    for positive_word in positives:   # for all positive words
        if positive_word in tweet.translate(None, punctuation).lower():   # if positive word in tweet
            positive_word_dict[positive_word] = positive_word_dict.get(positive_word, 0)  # WHAT DOES GET DO?
            positive_word_dict[positive_word] += 1
sorted(positive_word_dict.items())[:10]
Explanation: Find positive sentiment from tweets
<ul>
<li>Helpful to standardize tweets
<li>Convert to lower case
<li>Remove punctuation
</ul>
End of explanation
[item for item in Counter(positive_word_dict).most_common(1)]
Explanation: Find the most common positive word used in the tweets
End of explanation
## SOLUTION
negative = open("negative.txt").read()
negative_word_dict = {}
for tweet in tweets:
    for negative_word in negative.split('\n')[::-1][1:]:
        if negative_word in tweet.translate(None, punctuation).lower():
            negative_word_dict[negative_word] = negative_word_dict.get(negative_word, 0)
            negative_word_dict[negative_word] += 1
negative_counts = [item for item in Counter(negative_word_dict).most_common(10)]
print negative_counts
counter = 0
for tweet in tweets:
    if negative_counts[0][0] in tweet.translate(None, punctuation).lower():
        counter += 1
        if counter <= 10:
            print tweet
            print 80*'-'
        else:
            break
Explanation: ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
EXERCISE TIME!
Do a similar analysis above but using negative words. 
1) Read in the negative.txt file and store it in a variable call it negative.<br>
2) Using negative, find the 10 most common negative words in the tweets. Comment on any interesting observations.<br>
3) For the most common negative word, print 10 tweets that contain it. Comment on any interesting observations.
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
End of explanation
from re import match, search, findall, compile, sub
# python treats / as an escape character. So "/n" is a newline object, not a literal '/n'
# to de-pythonify a string, add r' to the front - this makes it a raw string
string = 'This is a\nnormal string with a newline and \ttab character'
rawString = r'and this is a\nraw string which can be used to search for \tpatterns'
print string
print "+++++++++++++++++"
print rawString
Explanation: What happened??
Regular Expression <a id="regex"></a>
<ul>
<li>Special sequence of characters that helps you match (**exact!**) or find other strings or sets of strings, using a specialized syntax held in a pattern
<li>Regular expressions as a concept is not exclusive to Python at all.
<li>Regular expressions are widely used in UNIX world.
<li>`re` module
        <ul>
        <li>`re.compile()`
        <li>`re.match()`
        <li>`re.search()`
        <li>`re.findall()`
        <li>`re.sub()`
        </ul>
<li>In python, regular expressions tend to be **very slow**. So only use if necessary.
</ul>
End of explanation
test_string = "I'm learning about regular expressions and about how to use them in Python. This is fantastic stuff!"
match_pattern = match("I'm", test_string) # return pattern matched by query
print match_pattern
print match_pattern.group()
print match("about", test_string)
Explanation: match - only find matches if they occur at the start of the string being searched.
End of explanation
search_pattern = search('about', test_string)
print search_pattern
search_pattern.group()
Explanation: search - similar to match, but search doesn’t restrict us to only finding matches at the beginning of the string. But will only return the first result
End of explanation
findall('about',test_string)
findall('an',test_string)  # no good
findall(' an ',test_string)  # kinda hacky
word = 'about'
findall(r'\b' + word + r'\b', test_string)  # better
word = 'about'
test_string.count(word)  # count() is ok for simple searches
Explanation: findall - get a list of all matching patterns
End of explanation
word = 'about'
expr = compile(r'\b' + word + r'\b')  # can get complex
print findall(expr, test_string)
Explanation: Use compile to create regular expressions that you can then search, findall, match, ...
End of explanation
## SOLUTION
negative = open("negative.txt").read()
negative_word_dict = {}
for tweet in tweets:
    for negative_word in negative.split('\n')[::-1][1:]:  # Since the last item is a ' ', reverse the list and skip it
     
    #    Note that you could also use count() - this is much faster
        num_occurence = tweet.translate(None, punctuation).lower().count(' ' + negative_word + ' ')
    
        # Unlike the previos exercise - here we increment the dictionary by the number of occurences, 
        # not just 1 for each occurence
    #    num_occurence = len(findall(r'\b' + negative_word + r'\b', tweet.translate(None, punctuation).lower()))
        
        if num_occurence > 0:
            negative_word_dict[negative_word] = negative_word_dict.get(negative_word, 0)
            negative_word_dict[negative_word] += num_occurence
most_negative = [item for item in Counter(negative_word_dict).most_common(10)]
print most_negative
counter = 0
# store most negative word as a regular expression
regex = compile(r'\s' + most_negative[0][0] + '\s')
for tweet in tweets:
    # search returns a true or false
    if search(regex, tweet.translate(None, punctuation).lower()):
        counter += 1
        if counter <= 10:
            print tweet
            print 80*'-'   # add dashed line for easier reading
        else:
            break
Explanation: ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
EXERCISE TIME!
Redo the negative tweet exercise from above to perform exact matching. That is:<br>
1) Using negative, find the 10 most common negative words in the tweets. Comment on any interesting observations.<br>
2) For the most common negative word, print 10 tweets that contain it. Comment on any interesting observations.
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
End of explanation
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
common_words = Counter(positive_word_dict).most_common(40)
common_words
ys = [i[1] for i in common_words]   # number of occurrences
xs = [i[0] for i in common_words]   # words
plt.figure(figsize=(18,8))  # set the canvas size
plt.plot(ys, linewidth = 2)    # call a line plt
pass
import seaborn as sns   # let's use seaborn in our plotting environment
plt.figure(figsize=(18,8))  # set the canvas size
plt.plot(ys, linewidth = 2)    # call a line plt
pass
Explanation: What is the distribution of negative word occurrence?
Let's plot it!
End of explanation
plt.figure(figsize=(18,8))  # set the canvas size
plt.plot(ys, linewidth = 2)    # call a line plt
plt.xticks(range(len(xs)), xs, rotation = 40)
pass
Explanation: Is this chart helpful?
Where are the words??
End of explanation
word_length_dict = {}
for tweet in tweets:
    for word in filter(None,tweet.translate(None, punctuation).split(' ')):
        word_length_dict[len(word)] = word_length_dict.get(len(word),0) + 1
word_length_dict
ys = [v for k, v in word_length_dict.items()]
xs = [k for k, v in word_length_dict.items()]
plt.figure(figsize=(18,8))
plt.plot(ys, linewidth=2)
plt.xticks(range(len(xs)), xs,rotation=35)
plt.title("Distribution of word length",size=16)
plt.ylabel("Frequency",size=14)
plt.xlabel("Word length",size=14)
pass
Explanation: Distribution of word length
End of explanation
for i in range(max(word_length_dict.keys())):
    if i not in word_length_dict.keys():
        word_length_dict[i] = 0
word_length_dict
ys = [v for k, v in word_length_dict.items()]
xs = [k for k, v in word_length_dict.items()]
plt.figure(figsize=(18,8))
plt.plot(ys,linewidth=2)
plt.xticks(range(len(xs)), xs,rotation=35)
plt.title("Distribution of word length",size=16)
plt.ylabel("Frequency",size=14)
plt.xlabel("Word length",size=14)
pass
Explanation: Anyone notice something strange about the chart?
End of explanation
# example of bi grams of first tweet
tweet = tweets[0].split(' ')
bigrams = []
for i in range(len(tweet)-1): # need to stop 1 element before the end
    bigrams.append((tweet[i], tweet[i+1]))
bigrams
# example of trigrams of first tweet
tweet = tweets[0].split(' ')
trigrams = []
for i in range(len(tweet)-2):
    trigrams.append((tweet[i], tweet[i+1], tweet[i+2]))
trigrams
Explanation: n-grams
<ul>
<li>Single word counts is ok, but for richer text analysis might want combinations of words
<li>bi-grams, tri-grams, ..., n-grams
</ul>
End of explanation
## SOLUTION
def bigrams(twt):
    return zip(twt, twt[1:])
print bigrams(tweet)
print
def trigrams(twt):
    return zip(twt, twt[1:], twt[2:])
print trigrams(tweet)
Explanation: +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
EXERCISE TIME
Write your own bigram function. Do this without using a for loop. Hint - look up the zip function.
Do the same for tri grams.
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
End of explanation
## SOLUTION
def remove_vowels(s):
    vowels = "aeiouAEIOU"
    s_without_vowels = ""
    for letter in s:
        if letter not in vowels:
            s_without_vowels += letter
    return s_without_vowels
remove_vowels('This sentence has no vowels. Cray cray!')
Explanation: Take Away
<ul>
<li>Manipulating strings and text analysis
<li>Reading files from the web and local 
<li>Basic plotting
</ul>
Review Problems<a id="hmwk"></a>
Q1.
Write a function to remove vowels from a string. Stick to base python only.
End of explanation
import os
if not os.path.exists('alice.txt'):
    ! curl http://www.gutenberg.org/cache/epub/12/pg12.txt -O alice.txt        
poem = open('pg12.txt').read()  # read in file
print poem[:1000]
Explanation: Q2.
For the next few questions, read in Through the Looking glass from Project Gutenberg (pg12.txt).
End of explanation
start = poem.find('JABBERWOCKY')
end = poem.find('It seems very pretty', start, end)
poem = poem[start:end - 1]
Explanation: A). Print the subset starting from 'JABBERWOCKY' and ending with 'It seems very pretty.'
End of explanation
## SOLUTION
import string
poem = poem.lower()
poem = poem.translate(None, string.punctuation)
words = poem.split()
def is_palindrome(word):
    return word == word[::-1] 
print set(word for word in words if is_palindrome(word))
Explanation: B). Print all distinct palindromic words, if any. Remember to 'standardize' first.
End of explanation
## SOLUTION 1
list(zip(words[:-2], words[1:-1], words[2:]))[:10]
## SOLUTION 2
import itertools
def window(x, n):
    Sliding widnow of size n from iterable x.
    s = (itertools.islice(x, i, None) for i in range(n))
    return zip(*s)
list(window(words, 3))[:10]
Explanation: C). Find all possible sequences of 3 words in the poem. This is called a trigram. (Hint: use zip or islice from itertools)
End of explanation
## SOLUTION
import pandas as pd
df = pd.DataFrame.from_dict(positive_word_dict,orient = 'index')
df.columns = ['num']
df.sort_values(by = ['num'], inplace = True, ascending = False)
plt.figure(figsize=(10,4))
plt.plot(df.num.values[:30],linewidth=3,color='red')
plt.xticks(range(len(df.index.values))[:30], df.index.values[:30],rotation=35)
plt.title("Top 30 positive word occurrence",size=16)
plt.ylabel("Frequency",size=14)
plt.xlabel("Positive words",size=14)
pass
df = pd.DataFrame.from_dict(positive_word_dict,orient = 'index')
df.columns = ['num']
df.sort_values(by = ['num'], inplace = True, ascending = False)
df['cumulative_proportion'] = np.cumsum(df.num).values/sum(df.num.values)
plt.figure(figsize=(10,4))
plt.plot(df['cumulative_proportion'].values[:30],linewidth=3,color='red')
plt.xticks(range(len(df.index.values))[:30], df.index.values[:30],rotation=35)
plt.title("Top 30 positive word occurrence",size=16)
plt.ylabel("Frequency",size=14)
plt.xlabel("Positive words",size=14)
pass
Explanation: Q3.
Plot a similar distribution for the positive word count. The differences are:
<ul>
<li>Plot top 30 positive words only
<li>Change the figure size
<li>Change the rotation of the xticks
<li>Change the linewidth and color
</ul>
Feel free to use the existing code as a template.<br>
As an extra challenge, instead of plotting the count of each positive word in ascending order, plot the cumulative count of the positive words, as a % of the total count of positive words, for the first 30. Your chart should look something like this:
<img id="cumulative" src="../images/cumulative.png"/>
End of explanation
## SOLUTION
import string
def encode(poem, shift):
    
    shifted_alphabet = string.ascii_lowercase[shift:] + string.ascii_lowercase[:shift]
    table = string.maketrans(string.ascii_lowercase, shifted_alphabet)
    
    return poem.translate(table)
# Encode
cipher = encode(poem, 2)
print(cipher)
# Decode
recovered = encode(cipher, -2)
print(recovered)
Explanation: BONUS QUESTION!
Q1. Encode and decode Through the Looking Glass using a Caesar cipher <br>
Hint: create a function encode(poem, shift) where shift is the number of characters used to encode the cipher. It should return the encoded poem. Can you also use this function to return the decoded cipher? Check out the maketrans function in string
End of explanation
<END_TASK>