<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Python NLTK
Step1:  The optional parameters in the open function above define the mode of operations on the file and the encoding of the content. For example, setting the mode to r declares that reading from the file is the only permitted operation that we will perform in the following code. Setting the encoding to utf-8 declares that all characters will be encoded using the Unicode encoding schema UTF-8 for the content of the file.
We can now import the NLTK module in Python to work with frequency profiles and n-grams using the tokens or words in the text.
Step2:  We can now lower the text, which means normalizing it to all characters lower case
Step3:  To generate a frequency profile from the text file, we can use the NLTK function FreqDist
Step4:  We can remove certain characters from the distribution, or alternatively replace these characters in the text variable. The following loop removes them from the frequency profile in myFD, which is a dictionary data structure in Python.
Step5:  We can print out the frequency profile by looping through the returned data structure
Step6:  To relativize the frequencies, we need to compute the total number of characters. This is assuming that we removed all punctuation symbols. The frequency distribution instance myFD provides a method to access the values associated with the individual characters. This will return a list of values, that is the frequencies associated with the characters.
Step7:  The sum function can summarize these values in its list argument
Step8:  To avoid type problems when we compute the relative frequency of characters, we can convert the total number of characters into a float. This will guarantee that the division in the following relativisation step will be a float as well.
Step9:  We store the resulting number of characters in the total variable
Step10:  We can now generate a probability distribution over characters. To convert the frequencies into relative frequencies we use list comprehension and divide every single list element by total. The resulting relative frequencies are stored in the variable relfreq
Step11:  Let us compute the Entropy for the character distribution using the relative frequencies. We will need the logarithm function from the Python math module for that
Step12:  We can define the Entropy function according to the equation $I = - \sum P(x) log_2( P(x) )$ as
Step13:  We can now compute the entropy of the character distribution
Step14:  We might be interested in the point-wise entropy of the characters in this distribution, thus needing the entropy of each single character. We can compute that in the following way
Step15:  We could now compute the variance over this point-wise entropy distribution or other properties of the frequency distribution as for example median, mode, or standard deviation.
From Characters to Words/Tokens
We see that the frequency profile is for the characters in the text, not the words or tokens. In order to generate a frequency profile over words/tokens in the text, we need to utilize a tokenizer. NLTK provides basic tokenization functions. We will use the word_tokenize function to generate a list of tokens
Step16:  We can print out the first 20 tokens to verify our data structure is a list with lower-case strings
Step17:  We can now generate a frequency profile from the token list, as we did with the characters above
Step19:  The frequency profile can be printed out in the same way as above by looping over the tokens and their frequencies. Note that we restrict the loop to the first 20 tokens here just to keep the notebook smaller. You can remove the [
Step20:  Counting N-grams
NLTK provides simple methods to generate n-gram models or frequency profiles over n-grams from any kind of list or sequence. We can for example generate a bi-gram model, that is an n-grams model for n = 2, from the text tokens
Step21:  To store the bigrams in a list that we want to process and analyze further, we convert the Python generator object myTokenBigrams to a list
Step22:  Let us verify that the resulting data structure is indeed a list of string tuples. We will print out the first 20 tuples from the bigram list
Step23:  We can now verify the number of bigrams and check that there are exactly number of tokens - 1 = number of bigrams in the resulting list
Step24:  The frequency profile from these bigrams is generated in exactly the same way as from the token list in the examples above
Step25:  If we would want to know some more general properties of the frequency distribution, we can print out information about it. The print statement for this bigram frequency distribution tells us that we have 17,766 types and 38,126 tokens
Step26:  The bigrams and their corresponding frequencies can be printed using a for loop. We restrict the number of printed items to 20, just to keep this list reasonably long. If you would like to see the full frequency profile, remove the [
Step27:  Pretty printing the bigrams is possible as well
Step28:  You can remove the [
Step29:  We can generate an increasing frequency profile using the sort function on the second element of the tuple list, that is on the frequency
Step30:  We can increase the speed of this sorted call by using the itemgetter() function in the operator module. Let us import this function
Step31:  We can now define the sort-key for sorted using the itemgetter function and selecting with 1 the second element in the tuple. Remember that the enumeration of elements in lists or tuples in Python starts at 0.
Step32:  A decreasing frequency profile can be generated using another parameter to sorted
Step33:  We can pretty-print the decreasing frequency profile
<ASSISTANT_TASK:>
Python Code:
ifile = open("data/HOPG.txt", mode='r', encoding='utf-8')
text = ifile.read()
ifile.close()
print(text[:300], "...")
Explanation: Python NLTK: Texts and Frequencies
(C) 2017-2019 by Damir Cavar <&#100;&#99;&#97;&#118;&#97;&#114;&#64;&#105;&#117;&#46;&#101;&#100;&#117;>
Version: 0.5, October 2019
Download: This and various other Jupyter notebooks are available from my GitHub repo.
License: Creative Commons Attribution-ShareAlike 4.0 International License (CA BY-SA 4.0)
This is a brief introduction to NLTK for simple frequency analysis of texts. I created this notebook for intro to corpus linguistics and natural language processing classes at Indiana University between 2017 and 2019.
For this to work, in the folder with the notebook we expect a subfolder data that contains a file HOPG.txt. This file contains the novel "A House of Pomegranates" by Oscar Wilde taken as raw text from Project Gutenberg.
Simple File Processing
Reading a text into memory in Python is faily simple. We open a file, read from it, and close the file again. The following code prints out the first 300 characters of the text in memory:
End of explanation
import nltk
Explanation: The optional parameters in the open function above define the mode of operations on the file and the encoding of the content. For example, setting the mode to r declares that reading from the file is the only permitted operation that we will perform in the following code. Setting the encoding to utf-8 declares that all characters will be encoded using the Unicode encoding schema UTF-8 for the content of the file.
We can now import the NLTK module in Python to work with frequency profiles and n-grams using the tokens or words in the text.
End of explanation
text = text.lower()
print(text[:300], "...")
Explanation: We can now lower the text, which means normalizing it to all characters lower case:
End of explanation
myFD = nltk.FreqDist(text)
print(myFD)
Explanation: To generate a frequency profile from the text file, we can use the NLTK function FreqDist:
End of explanation
for x in ":,.-[];!'\"\t\n/ ?":
    del myFD[x]
Explanation: We can remove certain characters from the distribution, or alternatively replace these characters in the text variable. The following loop removes them from the frequency profile in myFD, which is a dictionary data structure in Python.
End of explanation
for x in myFD:
    print(x, myFD[x])
Explanation: We can print out the frequency profile by looping through the returned data structure:
End of explanation
myFD.values()
Explanation: To relativize the frequencies, we need to compute the total number of characters. This is assuming that we removed all punctuation symbols. The frequency distribution instance myFD provides a method to access the values associated with the individual characters. This will return a list of values, that is the frequencies associated with the characters.
End of explanation
sum(myFD.values())
Explanation: The sum function can summarize these values in its list argument:
End of explanation
float(sum(myFD.values()))
Explanation: To avoid type problems when we compute the relative frequency of characters, we can convert the total number of characters into a float. This will guarantee that the division in the following relativisation step will be a float as well.
End of explanation
total = float(sum(myFD.values()))
print(total)
Explanation: We store the resulting number of characters in the total variable:
End of explanation
relfrq = [ x/total for x in myFD.values() ]
print(relfrq)
Explanation: We can now generate a probability distribution over characters. To convert the frequencies into relative frequencies we use list comprehension and divide every single list element by total. The resulting relative frequencies are stored in the variable relfreq:
End of explanation
from math import log
Explanation: Let us compute the Entropy for the character distribution using the relative frequencies. We will need the logarithm function from the Python math module for that:
End of explanation
def entropy(p):
    return -sum( [ x * log(x, 2) for x in p ] )
entropy([1/8, 1/16, 1/4, 1/8, 1/16, 1/16, 1/4, 1/16])
Explanation: We can define the Entropy function according to the equation $I = - \sum P(x) log_2( P(x) )$ as:
End of explanation
print(entropy(relfrq))
Explanation: We can now compute the entropy of the character distribution:
End of explanation
entdist = [ -x * log(x, 2) for x in relfrq ]
print(entdist)
Explanation: We might be interested in the point-wise entropy of the characters in this distribution, thus needing the entropy of each single character. We can compute that in the following way:
End of explanation
tokens = nltk.word_tokenize(text)
Explanation: We could now compute the variance over this point-wise entropy distribution or other properties of the frequency distribution as for example median, mode, or standard deviation.
From Characters to Words/Tokens
We see that the frequency profile is for the characters in the text, not the words or tokens. In order to generate a frequency profile over words/tokens in the text, we need to utilize a tokenizer. NLTK provides basic tokenization functions. We will use the word_tokenize function to generate a list of tokens:
End of explanation
tokens[:20]
Explanation: We can print out the first 20 tokens to verify our data structure is a list with lower-case strings:
End of explanation
myTokenFD = nltk.FreqDist(tokens)
print(myTokenFD)
Explanation: We can now generate a frequency profile from the token list, as we did with the characters above:
End of explanation
for token in list(myTokenFD.items()):
    print(token[0], token[1])
stopwords = 
i
me
my
myself
we
our
ours
ourselves
you
you're
you've
you'll
you'd
your
yours
yourself
yourselves
he
him
his
himself
she
she's
her
hers
herself
it
it's
its
itself
they
them
their
theirs
themselves
what
which
who
whom
this
that
that'll
these
those
am
is
are
was
were
be
been
being
have
has
had
having
do
does
did
doing
a
an
the
and
but
if
or
because
as
until
while
of
at
by
for
with
about
against
between
into
through
during
before
after
above
below
to
from
up
down
in
out
on
off
over
under
again
further
then
once
here
there
when
where
why
how
all
any
both
each
few
more
most
other
some
such
no
nor
not
only
own
same
so
than
too
very
s
t
can
will
just
don
don't
should
should've
now
d
ll
m
o
re
ve
y
ain
aren
aren't
couldn
couldn't
didn
didn't
doesn
doesn't
hadn
hadn't
hasn
hasn't
haven
haven't
isn
isn't
ma
mightn
mightn't
mustn
mustn't
needn
needn't
shan
shan't
shouldn
shouldn't
wasn
wasn't
weren
weren't
won
won't
wouldn
wouldn't
for x in stopwords.split():
    del myTokenFD[x]
print(list(myTokenFD))
Explanation: The frequency profile can be printed out in the same way as above by looping over the tokens and their frequencies. Note that we restrict the loop to the first 20 tokens here just to keep the notebook smaller. You can remove the [:20] selector in your own experiments.
End of explanation
myTokenBigrams = nltk.ngrams(tokens, 2)
Explanation: Counting N-grams
NLTK provides simple methods to generate n-gram models or frequency profiles over n-grams from any kind of list or sequence. We can for example generate a bi-gram model, that is an n-grams model for n = 2, from the text tokens:
End of explanation
bigrams = list(myTokenBigrams)
Explanation: To store the bigrams in a list that we want to process and analyze further, we convert the Python generator object myTokenBigrams to a list:
End of explanation
print(bigrams[:20])
Explanation: Let us verify that the resulting data structure is indeed a list of string tuples. We will print out the first 20 tuples from the bigram list:
End of explanation
print(len(bigrams))
print(len(tokens))
Explanation: We can now verify the number of bigrams and check that there are exactly number of tokens - 1 = number of bigrams in the resulting list:
End of explanation
myBigramFD = nltk.FreqDist(bigrams)
print(myBigramFD)
Explanation: The frequency profile from these bigrams is generated in exactly the same way as from the token list in the examples above:
End of explanation
print(myBigramFD)
Explanation: If we would want to know some more general properties of the frequency distribution, we can print out information about it. The print statement for this bigram frequency distribution tells us that we have 17,766 types and 38,126 tokens:
End of explanation
for bigram in list(myBigramFD.items())[:20]:
    print(bigram[0], bigram[1])
print("...")
Explanation: The bigrams and their corresponding frequencies can be printed using a for loop. We restrict the number of printed items to 20, just to keep this list reasonably long. If you would like to see the full frequency profile, remove the [:20] restrictor.
End of explanation
for ngram in list(myBigramFD.items()):
    print(" ".join(ngram[0]), ngram[1])
print("...")
Explanation: Pretty printing the bigrams is possible as well:
End of explanation
ngrams = [ (" ".join(ngram), myBigramFD[ngram]) for ngram in myBigramFD ]
print(ngrams[:100])
Explanation: You can remove the [:20] restrictor above and print out the entire frequency profile. If you select and copy the profile to your clipboard, you can paste it into your favorite spreadsheet software and sort, analyze, and study the distribution in many interesting ways.
Instead of running the frequency profile through a loop we can also use a list comprehension construction in Python to generate a list of tuples with the n-gram and its frequency:
End of explanation
sortedngrams = sorted(ngrams, key=lambda x: x[1])
print(sortedngrams[:20])
print("...")
Explanation: We can generate an increasing frequency profile using the sort function on the second element of the tuple list, that is on the frequency:
End of explanation
from operator import itemgetter
Explanation: We can increase the speed of this sorted call by using the itemgetter() function in the operator module. Let us import this function:
End of explanation
sortedngrams = sorted(ngrams, key=itemgetter(1))
print(sortedngrams[:20])
print("...")
Explanation: We can now define the sort-key for sorted using the itemgetter function and selecting with 1 the second element in the tuple. Remember that the enumeration of elements in lists or tuples in Python starts at 0.
End of explanation
sortedngrams = sorted(ngrams, key=itemgetter(1), reverse=True)
print(sortedngrams[:20])
print("...")
Explanation: A decreasing frequency profile can be generated using another parameter to sorted:
End of explanation
sortedngrams = sorted(ngrams, key=itemgetter(1), reverse=True)
for t in sortedngrams[:20]:
    print(t[0], t[1])
print("...")
total = float(sum(myBigramFD.values()))
exceptions = ["]", "[", "--", ",", ".", "'s", "?", "!", "'", "'ye"]
myStopWords = stopwords.split()
results = []
for x in myBigramFD:
    if x[0] in exceptions or x[1] in exceptions:
        continue
    if x[0] in myStopWords or x[1] in myStopWords:
        continue
    #print("%s\t%s\t%s" % (x[0], x[1], myBigramFD[x]/total))
    results.append( (x[0], x[1], myBigramFD[x]/total) )
#print(results)
sortedresults = sorted(results, key=itemgetter(2), reverse=True)
for x in sortedresults[:20]:
    print(x[0], x[1], x[2])
Explanation: We can pretty-print the decreasing frequency profile:
End of explanation
<END_TASK>