<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Counting Word Frequencies with Python
The debates that occur in the Canadian Parliament are transcribed and published in a document known as <i>Hansard</i>. Since 2006, <i>Hansard</i> has been available for public download in <code>.xml</code> file format. To date, this archive contains over 57 million words. By converting these transcripts from <code>.xml</code> to <code>.txt</code>, the files can be processed in numerous ways with the <i>Python</i> coding language.</p>
<p>The purpose of this document is to illustrate the counting of specific words in the collection of text files that will be referred to from now on as the <b><span style="cursor
Step1:  Now that we have loaded our file, we can begin to work on it. <i>Python</i> offers us a lot of pre-built tools to make the task of coding easier. Some of the most commonly used tools are known as <b><span style="cursor
Step2:  Now we can call the function for any word we choose. The next example shows that there are <u>392</u> occurences of the word <code>privacy</code> contained in the transcripts for the 39th sitting of Parliament.
Step3:  Unfortunately, there are two distinct problems here, centred around the fact that our function is only counting the string <code>privacy</code> exactly as it appears.</p>
<p>The first problem is that text strings are case-sensitive. If the word contains UPPERCASE and lowercase letters, the word that is searched for will only be counted if the cases match exactly. The following example counts the number of instances of <code>Privacy</code> with the first letter capitalized.
Step4:  Here is a more extreme example to illustrate the point.
Step5:  The second problem is that of punctuation. Much like words are case-sensitive, they are also punctuation-sensitive. If a piece of punctuation has been included in the string, it will be included in the search. Here we count the occurrences of <code>privacy,</code> shown here with a comma after the word.
Step6:  And here we count <code>privacy.</code>, with the word followed by a period.
Step7:  We could comb through the text to find all of the different instantiations of <code>privacy</code>, and then run the code for each one and add together all of the numbers, but that would be time consuming and potentially inaccurate. Instead, we must process the text further to make the text uniform. In this case we want to make all of the characters lowercase, and remove all of the punctuation.</p>
<p><i>Python</i> has a function that will do this for us.
We will reload the text file, split the text into distinct words or <b><span style="cursor
Step8:  Now, when we count the instances of <code>privacy</code>, we are presented with a total of <u>846</u> instances.
Step9:  Part 2
Step10:  Although we can display the contents of a directory by using the <code>listdir</code> function, <i>Python</i> needs those names stored in a list in order to iterate over it. We also want to specify that only files with the extension <code>.txt</code> are included. Here we create another function called <code>list_textfiles</code>.
Step11:  Rather than writing code to open each file individually, we can create another custom function to open the file we pass to it. We'll call this one read_file.
Step12:  Now we can open all of the files in our directory, strip each file of uppercase letters and punctuation, split the whole of each text into tokens, and store all the data as separate lists in our variable <code>corpus</code>.
Step13:  Let's check to make sure the code worked by using the <code>len</code> function to count the number of items in our <code>corpus</code> list.
Step14:  Let's create a function to make the names of the files more readable. First we'll have to strip the the file extension <code>.txt</code>.
Step15:  Now let's make a function to remove the <code>data/</code>.
Step16:  And finally, we'll write a function to tie the two functions together.
Step17:  Now we can display a readable list of the files within our directory.
Step18:  The next step involves iterating through both lists
Step19:  What's exciting about this code is that we can now search the entire corpus for any word we choose. Let's search for <code>information</code>.
Step20:  How about <code>ethics</code>?
<ASSISTANT_TASK:>
Python Code:
# 1. open the text file
infile = open('data/39.txt')
# 2. read the file and assign it to the variable 'text'
text = infile.read()
# 3. close the text file
infile.close()
# 4. split the variable 'text' into distinct word strings
words = text.split()
Explanation: Counting Word Frequencies with Python
The debates that occur in the Canadian Parliament are transcribed and published in a document known as <i>Hansard</i>. Since 2006, <i>Hansard</i> has been available for public download in <code>.xml</code> file format. To date, this archive contains over 57 million words. By converting these transcripts from <code>.xml</code> to <code>.txt</code>, the files can be processed in numerous ways with the <i>Python</i> coding language.</p>
<p>The purpose of this document is to illustrate the counting of specific words in the collection of text files that will be referred to from now on as the <b><span style="cursor:help;" title="a large and structured set of texts">corpus</b></span>. Terms that may be unfamiliar to the reader are highlighted in <b><span style="cursor:help;" title="hover the mouse over the word to read the definition">bold</span></b>, and contain a definition that can be accessed by hovering the mouse pointer over the word. While it is not necessary to read every line of code to understand the process, explanatory sections within the code are marked with a # and coloured light blue.
### Part 1: Determining the word frequency for a single file
Before we can begin working with a piece of text, it must be loaded into and read by <i>Python</i>. Rather than altering (and potentially irreversibly changing) our original text file, we will work with the contents of the file as a <b><span style="cursor:help;" title="any finite sequence of characters (i.e., letters, numerals, symbols and punctuation marks)">string</span></b> of text contained within a <b><span style="cursor:help;" title="a value that can change, depending on conditions or on information passed to the program">variable</span></b>. Now we can close our original text file, keeping it intact.
In the next piece of code we will open the file that contains the textual transcripts of all of the Parliamentary debates that occured in the House of Commons during the 39th sitting of Parliament.
While working with one long string may be useful for other applications, our purposes require that we split the text into pieces. In this case, each word will become it's own unique string.
End of explanation
# 5. define the'count_in_list' function
def count_in_list(item_to_count, list_to_search):
    "Counts the number of a specified word within a list of words"
    number_of_hits = 0
    for item in list_to_search:
        if item == item_to_count:
            number_of_hits += 1
    return number_of_hits
Explanation: Now that we have loaded our file, we can begin to work on it. <i>Python</i> offers us a lot of pre-built tools to make the task of coding easier. Some of the most commonly used tools are known as <b><span style="cursor:help;" title="a set of instructions that performs a specific task">functions</span></b>. Functions are useful for automating tasks that would otherwise require a repetitive amount of coding. While <i>Python</i> has many built-in functions, the language's true power comes from the ability to define unique functions based on programming needs. In the code above, we've already used four <i>Python</i> functions: <code>open</code>, <code>read</code>, <code>close</code>, and <code>split</code>.</p>
<p>In the next piece of code we will define our own funtion, called <code>count_in_list</code>. This function will allow us to count the occurence of any word in the corpus.
End of explanation
# 6. here the function counts the instances of the word 'privacy'
print "Instances of the word \'privacy\':", (count_in_list("privacy", words))
Explanation: Now we can call the function for any word we choose. The next example shows that there are <u>392</u> occurences of the word <code>privacy</code> contained in the transcripts for the 39th sitting of Parliament.
End of explanation
print "Instances of the word \'Privacy\':", (count_in_list("Privacy", words))
Explanation: Unfortunately, there are two distinct problems here, centred around the fact that our function is only counting the string <code>privacy</code> exactly as it appears.</p>
<p>The first problem is that text strings are case-sensitive. If the word contains UPPERCASE and lowercase letters, the word that is searched for will only be counted if the cases match exactly. The following example counts the number of instances of <code>Privacy</code> with the first letter capitalized.
End of explanation
print "Instances of the word \'pRiVaCy\':",(count_in_list("pRiVaCy", words))
Explanation: Here is a more extreme example to illustrate the point.
End of explanation
print "Instances of the word \'privacy,\':", (count_in_list("privacy,", words))
Explanation: The second problem is that of punctuation. Much like words are case-sensitive, they are also punctuation-sensitive. If a piece of punctuation has been included in the string, it will be included in the search. Here we count the occurrences of <code>privacy,</code> shown here with a comma after the word.
End of explanation
print "Instances of the word \'privacy.\':",(count_in_list("privacy.", words))
Explanation: And here we count <code>privacy.</code>, with the word followed by a period.
End of explanation
infile = open('data/39.txt')
text = infile.read()
infile.close()
tokens = text.split()
#here we call the text cleaning function
words = [w.lower() for w in tokens if w.isalpha()]
Explanation: We could comb through the text to find all of the different instantiations of <code>privacy</code>, and then run the code for each one and add together all of the numbers, but that would be time consuming and potentially inaccurate. Instead, we must process the text further to make the text uniform. In this case we want to make all of the characters lowercase, and remove all of the punctuation.</p>
<p><i>Python</i> has a function that will do this for us.
We will reload the text file, split the text into distinct words or <b><span style="cursor:help;" title="an individual occurence of a symbol or string">tokens</span></b> and then use the the text cleaning function.
End of explanation
print "Instances of the word \'privacy\':", (count_in_list("privacy", words))
Explanation: Now, when we count the instances of <code>privacy</code>, we are presented with a total of <u>846</u> instances.
End of explanation
# imports the os module
from os import listdir
# calls the listdir function to list the files in a specific directory
listdir("data")
Explanation: Part 2: Determining Word Frequencies for the Entire Corpus
Now let's see how this compares to the rest of the corpus. To accomplish this, we must write another function that will read all of the text files in our file folder.
<p>First we need to introduce a function of <i>Python</i> that we've yet to see: modules. Modules are packages of functions and code that serve specific purposes. These are much like functions, but more complex.
<p>The next piece of code imports a module called <code>os</code>, specifically the function <code>listdir</code>. We will use <code>listdir</code> to print a list of all the files in a specific directory. Each of the listed files corresponds to a textual transcript of Hansard. The first nine files refer to the complete transcript for each year from 2006 to 2014, while the last three files are the transcripts corresponding to each sitting of Parliament, in this case the 39th through to the end of the second sitting of the 41st Parliament.
End of explanation
def list_textfiles(directory):
    "Return a list of filenames ending in '.txt'"
    textfiles = []
    for filename in listdir(directory):
        if filename.endswith(".txt"):
            textfiles.append(directory + "/" + filename)
    return textfiles
Explanation: Although we can display the contents of a directory by using the <code>listdir</code> function, <i>Python</i> needs those names stored in a list in order to iterate over it. We also want to specify that only files with the extension <code>.txt</code> are included. Here we create another function called <code>list_textfiles</code>.
End of explanation
def read_file(filename):
    "Read the contents of FILENAME and return as a string."
    infile = open(filename)
    contents = infile.read()
    infile.close()
    return contents
Explanation: Rather than writing code to open each file individually, we can create another custom function to open the file we pass to it. We'll call this one read_file.
End of explanation
corpus = []
for filename in list_textfiles("data"):
    # reads the file
    text = read_file(filename)
    # splits the text into tokens
    tokens = text.split()
    # removes the punctuation and changes Uppercase to lower
    words = [w.lower() for w in tokens if w.isalpha()]
    # creates a set of word lists for each file
    corpus.append(words)
Explanation: Now we can open all of the files in our directory, strip each file of uppercase letters and punctuation, split the whole of each text into tokens, and store all the data as separate lists in our variable <code>corpus</code>.
End of explanation
print"There are", len(corpus), "files in the list, named: ", ', '.join(list_textfiles('data'))
Explanation: Let's check to make sure the code worked by using the <code>len</code> function to count the number of items in our <code>corpus</code> list.
End of explanation
from os.path import splitext
def remove_ext(filename):
    "Removes the file extension, such as .txt"
    name, extension = splitext(filename)
    return name
for files in list_textfiles('data'):
    remove_ext(files)
Explanation: Let's create a function to make the names of the files more readable. First we'll have to strip the the file extension <code>.txt</code>.
End of explanation
from os.path import basename
def remove_dir(filepath):
    "Removes the path from the file name"
    name = basename(filepath)
    return name
for files in list_textfiles('data'):
    remove_dir(files)
Explanation: Now let's make a function to remove the <code>data/</code>.
End of explanation
def get_filename(filepath):
    "Removes the path and file extension from the file name"
    filename = remove_ext(filepath)
    name = remove_dir(filename)
    return name
filenames = []
for files in list_textfiles('data'):
    files = get_filename(files)
    filenames.append(files)
Explanation: And finally, we'll write a function to tie the two functions together.
End of explanation
print"There are", len(corpus), "files in the list, named:", ', '.join(filenames),"."
Explanation: Now we can display a readable list of the files within our directory.
End of explanation
for words, names in zip(corpus, filenames):
    print"Instances of the word \'privacy\' in",names, ":", count_in_list("privacy", words)
Explanation: The next step involves iterating through both lists: <code>corpus</code> and <code>filenames</code>, in order to generate a word frequency for each file in the corpus. For this we will use <i>Python's</i> <code>zip</code> function.
End of explanation
for words, names in zip(corpus, filenames):
    print"Instances of the word \'information\' in",names, ":", count_in_list("information", words)
Explanation: What's exciting about this code is that we can now search the entire corpus for any word we choose. Let's search for <code>information</code>.
End of explanation
for words, names in zip(corpus, filenames):
    print"Instances of the word \'ethics\' in",names, ":", count_in_list("ethics", words)
Explanation: How about <code>ethics</code>?
End of explanation
<END_TASK>