<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 CRTC Hearing Text Analysis
The purpose of this notebook is to illustrate the method of text analysis using a corpus created from digital content published by the CRTC. This is the second part in a two-part process, the first of which is a description of the code that 'scraped' the CRTC webpage to create the corpus. 
Setting Up
The code below imports the modules that are required to process the text.
Step1:  Reading the File
This code loads and then reads the necessary files
Step2:  A bit of error checking here to confirm the number of records in the file. We should have 14.
Step3:  Changing the number in the code below will print a different record from the file. Please remember that in coding, numbered lists begin at 0.
Step4:  Here is a bit more error checking to confirm the record titles and their urls.
Step5:  And a bit more processing to make the text more readable. It's printed below.
Step6:  Text Analysis Processing
This is the begining of the first processing for the text analysis. Here we will split all the words apart, make them all lowercase, and remove the punctuation, numbers, and words on the stopword list.
Step7:  Since a word of interest is guarantee, here is a list of how many times that word (and its variations) appear in each record.
Step8:  Concordance
It looks like record number 5 has the most occurences of the word guarantee. The code below isolates the record and creates a concordance based on the selected word.
Step9:  Below is what the text looks like after the initial processing, without punctuation, numbers, or stopwords.
Step10:  Lemmatization
Some more preparation for the text processing. The code below works on the all of the records, creating one master list of words which is then lemmatized.
Step11:  Here we are checking to make sure the lemmatizer has worked. Now the word guarantee only appears in one form.
Step12:  Word Frequency
Here is a count of the number of words in each record. While this data isn't terribly useful 'as is', we can make a few assumptions about the text here. Notably that some of the hearings were much longer than others.
Step13:  Here we will count the five most common words in each record.
Step14:  These are the 10 most common word pairs in the text.
Step15:  Error checking to make sure the code is processing the text properly.
Step16:  More error checking.
Step17:  Below is a frequency plot showing the occurence of the 25 most frequent words.
Step18:  Collocations
Here we are preparing the text to search for bigrams containing the word guarantee. This code searches for words appearing before and after guarantee with a window size of two words on either side.
Step19:  Research shows that this is the most reliable statistical test for unreliable data. 
Log-Likelihood Ratio
The Log-likelihood ratio calculates the size and significance between the observed and expected frequencies of bigrams and assigns a score based on the result, taking into account the overall size of the corpus. The larger the difference between the observed and expected, the higher the score, and the more statistically significant the collocate is.
The Log-likelihood ratio is my preferred test for collocates because it does not rely on a normal distribution, and for this reason, it can account for sparse or low frequency bigrams. It does not over-represent low frequency bigrams with inflated scores, as the test is only reporting how much more likely it is that the frequencies are different than they are the same. The drawback to the Log-likelihood ratio is that it cannot be used to compare scores across corpora.
An important note here that words will appear twice in the following list. As the ngrams can appear both before and after the word, care must be taken to identify duplicate occurences in the list below and then combine the totals.
Step20:  Here is an example of words appearing twice. Below are both instances of the ngram 'quality'. The first instance appears before 'guarantee' and the second occurs after.
A bit more processing to clean up the list.
Step21:  Here is a list showing only the collocates for the word guarantee. Again, watch for duplicate words below.
Step22:  
<ASSISTANT_TASK:>
Python Code:
# importing code modules
import json
import ijson
from ijson import items
import pprint
from tabulate import tabulate
import matplotlib.pyplot as plt
import re
import csv
import sys
import codecs
import nltk
import nltk.collocations
import collections
import statistics
from nltk.metrics.spearman import *
from nltk.collocations import *
from nltk.stem import WordNetLemmatizer
# This is a function for reading the contents of files
def read_file(filename):
    "Read the contents of FILENAME and return as a string."
    infile = codecs.open(filename, 'r', 'utf-8')
    contents = infile.read()
    infile.close()
    return contents
Explanation: CRTC Hearing Text Analysis
The purpose of this notebook is to illustrate the method of text analysis using a corpus created from digital content published by the CRTC. This is the second part in a two-part process, the first of which is a description of the code that 'scraped' the CRTC webpage to create the corpus. 
Setting Up
The code below imports the modules that are required to process the text.
End of explanation
# loading the JSON file
filename = "../scrapy/hearing_result6.json"
# loading the stopwords file
stopwords = read_file('cornellStopWords.txt')
customStopwords = stopwords.split()
# reads the file and assigns the keys and values to a Python dictionary structure
with open(filename, 'r') as f:
    objects = ijson.items(f, 'item')
    file = list(objects)
Explanation: Reading the File
This code loads and then reads the necessary files: the json file with all the hearing text, and a txt file with a list of stopwords, taken from here: http://www.lextek.com/manuals/onix/stopwords2.html. I've also added a few custom words.
End of explanation
# checks to see how many records we have
print(len(file))
Explanation: A bit of error checking here to confirm the number of records in the file. We should have 14.
End of explanation
# commenting this out to make the github notebook more readable.
# prints all content in a single record. Changing the number shows a different record
file[0]
Explanation: Changing the number in the code below will print a different record from the file. Please remember that in coding, numbered lists begin at 0.
End of explanation
# iterates through each record in the file
for row in file:
    # prints the title of each record and its url
    print(row['title'], ":", row['url'])
Explanation: Here is a bit more error checking to confirm the record titles and their urls.
End of explanation
# appends all of the text items to a single string object (rather than a list)
joined_text = []
for row in file:
    joined_text.append(' '.join(row['text']))
# shows the text. Changing the number displays a different record...
# ...changing/removing the second number limits/expands the text shown.
print(joined_text[5][:750])
Explanation: And a bit more processing to make the text more readable. It's printed below.
End of explanation
# splits the text string in each record into a list of separate words
token_joined = []
for words in joined_text:
    # splits the text into a list of words
    text = words.split()
    # makes all words lowercase
    clean = [w.lower() for w in text if w.isalpha()]
    # applies stopword removal
    text = [w for w in clean if w not in customStopwords]
    token_joined.append(text)
Explanation: Text Analysis Processing
This is the begining of the first processing for the text analysis. Here we will split all the words apart, make them all lowercase, and remove the punctuation, numbers, and words on the stopword list.
End of explanation
#for title,word in zip(file,token_joined):
   # print(title['title'],"guarantee:", word.count('guarantee'), "guarantees:", \
       #   word.count('guarantees'), "guaranteed:", word.count('guaranteed'))
for title,word in zip(file,token_joined):
    print(title['title'],"service:", word.count('service'),"services:", word.count('services'))
Explanation: Since a word of interest is guarantee, here is a list of how many times that word (and its variations) appear in each record.
End of explanation
# splits the text from the record into a list of individual words
words = joined_text[0].split()
#assigns NLTK functionality to the text
text = nltk.Text(words)
# prints a concordance output for the selected word (shown in green)
print(text.concordance('services', lines=25))
#creates a new file that can be written by the print queue
fileconcord = codecs.open('April11_service_concord.txt', 'w', 'utf-8')
#makes a copy of the empty print queue, so that we can return to it at the end of the function
tmpout = sys.stdout
#stores the text in the print queue
sys.stdout = fileconcord
#generates and prints the concordance, the number pertains to the total number of bytes per line
text.concordance("service", 79, sys.maxsize)
#closes the file
fileconcord.close()
#returns the print queue to an empty state
sys.stdout = tmpout
Explanation: Concordance
It looks like record number 5 has the most occurences of the word guarantee. The code below isolates the record and creates a concordance based on the selected word.
End of explanation
# shows the text list for a given record. Changing the first number displays a... 
# ...different record, changing/removing the second number limits/expands the text shown
print(token_joined[5][:50])
Explanation: Below is what the text looks like after the initial processing, without punctuation, numbers, or stopwords.
End of explanation
# creates a variable for the lemmatizing function
wnl = WordNetLemmatizer()
# lemmatizes all of the verbs
lemm = []
for record in token_joined:
        for word in record:
            lemm.append(wnl.lemmatize(word, 'v'))
'''
lemm = []
for word in token_joined[13]:
        lemm.append(wnl.lemmatize(word, 'v'))
'''
# lemmatizes all of the nouns 
lems = []
for word in lemm:
    lems.append(wnl.lemmatize(word, 'n'))
Explanation: Lemmatization
Some more preparation for the text processing. The code below works on the all of the records, creating one master list of words which is then lemmatized.
End of explanation
# just making sure the lemmatizer has worked
#print("guarantee:", lems.count('guarantee'), "guarantees:", \
         # lems.count('guarantees'), "guaranteed:", lems.count('guaranteed'))
print("service:", lems.count('service'), lems.count('services'))
Explanation: Here we are checking to make sure the lemmatizer has worked. Now the word guarantee only appears in one form.
End of explanation
# counting the number of words in each record 
for name, each in zip(file,token_joined):
    print(name['title'], ":",len(each), "words")
Explanation: Word Frequency
Here is a count of the number of words in each record. While this data isn't terribly useful 'as is', we can make a few assumptions about the text here. Notably that some of the hearings were much longer than others.
End of explanation
docfreq = []
for words in token_joined:
    docfreq.append(nltk.FreqDist(words))
for name, words in zip(file_obj, docfreq):
    print(name['title'], ":", words.most_common(5))
Explanation: Here we will count the five most common words in each record.
End of explanation
# prints the 10 most common bigrams
colText = nltk.Text(lems)
colText.collocations(10)
Explanation: These are the 10 most common word pairs in the text.
End of explanation
# creates a list of bigrams (ngrams of 2), printing the first 5
colBigrams = list(nltk.ngrams(colText, 2)) 
colBigrams[:5]
Explanation: Error checking to make sure the code is processing the text properly.
End of explanation
# error checking. There should be one less bigram than total words
print("Number of words:", len(lems))
print("Number of bigrams:", len(colBigrams))
Explanation: More error checking.
End of explanation
# frequency plot with stopwords removed
%matplotlib inline
plt.rcParams['figure.figsize'] = (10.0, 10.0)
fd = nltk.FreqDist(colText)
fd.plot(25)
Explanation: Below is a frequency plot showing the occurence of the 25 most frequent words.
End of explanation
# loads bigram code from NLTK
bigram_measures = nltk.collocations.BigramAssocMeasures()
# bigrams with a window size of 2 words
finder = BigramCollocationFinder.from_words(lems, window_size = 2)
# ngrams with 'word of interest' as a member
word_filter = lambda *w: 'service' not in w
# only bigrams that contain the 'word of interest'
finder.apply_ngram_filter(word_filter)
# filter results based on statistical test
# calulates the raw frequency as an actual number and percentage of total words
act = finder.ngram_fd.items()
raw = finder.score_ngrams(bigram_measures.raw_freq)
# log-likelihood ratio
log = finder.score_ngrams(bigram_measures.likelihood_ratio)
Explanation: Collocations
Here we are preparing the text to search for bigrams containing the word guarantee. This code searches for words appearing before and after guarantee with a window size of two words on either side.
End of explanation
# prints list of results. 
print(tabulate(log, headers = ["Collocate", "Log-Likelihood"], floatfmt=".3f", \
               numalign="left"))
# prints list of results. 
print(tabulate(act, headers = ["Collocate", "Actual"], floatfmt=".3f", \
               numalign="left"))
with open('digital-literacy_collocate_Act.csv','w') as f:
    w = csv.writer(f)
    w.writerows(act)
Explanation: Research shows that this is the most reliable statistical test for unreliable data. 
Log-Likelihood Ratio
The Log-likelihood ratio calculates the size and significance between the observed and expected frequencies of bigrams and assigns a score based on the result, taking into account the overall size of the corpus. The larger the difference between the observed and expected, the higher the score, and the more statistically significant the collocate is.
The Log-likelihood ratio is my preferred test for collocates because it does not rely on a normal distribution, and for this reason, it can account for sparse or low frequency bigrams. It does not over-represent low frequency bigrams with inflated scores, as the test is only reporting how much more likely it is that the frequencies are different than they are the same. The drawback to the Log-likelihood ratio is that it cannot be used to compare scores across corpora.
An important note here that words will appear twice in the following list. As the ngrams can appear both before and after the word, care must be taken to identify duplicate occurences in the list below and then combine the totals.
End of explanation
##################################################################
############### sorts list of log-likelihood scores ##############
##################################################################
# group bigrams by first and second word in bigram                                        
prefix_keys = collections.defaultdict(list)
for key, l in log:
    # first word
    prefix_keys[key[0]].append((key[1], l))
    # second word
    prefix_keys[key[1]].append((key[0], l))
    
# sort bigrams by strongest association                                  
for key in prefix_keys:
    prefix_keys[key].sort(key = lambda x: -x[1])
    # prints top 80 results
logkeys = prefix_keys['service'][:80]
Explanation: Here is an example of words appearing twice. Below are both instances of the ngram 'quality'. The first instance appears before 'guarantee' and the second occurs after.
A bit more processing to clean up the list.
End of explanation
from tabulate import tabulate
print(tabulate(logkeys, headers = ["Collocate", "Log-Likelihood"], floatfmt=".3f", \
               numalign="left"))
with open('service_collocate_Log.csv','w') as f:
    w = csv.writer(f)
    w.writerows(logkeys)
Explanation: Here is a list showing only the collocates for the word guarantee. Again, watch for duplicate words below.
End of explanation
# working on a regex to split text by speaker
diced = []
for words in joined_text:
    diced.append(re.split('(\d+(\s)\w+[A-Z](\s|.\s)\w+[A-Z]:\s)', words))
print(diced[8])
init_names = []
for words in joined_text:
    init_names.append(set(re.findall('[A-Z]{3,}', words)))
print(init_names)
with open('initialNames.csv','w') as f:
    w = csv.writer(f)
    w.writerows(init_names)
Explanation: 
End of explanation
<END_TASK>