<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Introduction to NLTK
We have seen how to do some basic text processing in Python, now we introduce an open source framework for natural language processing that can further help to work with human languages
Step1:  Tokens
The basic atomic part of each text are the tokens. A token is the NLP name for a sequence of characters that we want to treat as a group.
We have seen how we can extract tokens by splitting the text at the blank spaces.
NTLK has a function word_tokenize() for it
Step2:  21 tokens extracted, which include words and punctuation.
Note that the tokens are different than what a split by blank spaces would obtained, e.g. "can't" is by NTLK considered TWO tokens
Step3:  And we can apply it to an entire book, "The Prince" by Machiavelli that we used last time
Step4:  As mentioned above, the NTLK tokeniser works in a more sophisticated way than just splitting by spaces, therefore we got this time more tokens.
Sentences
NTLK has a function to tokenise a text not in words but in sentences.
Step5:  As you see, it is not splitting just after each full stop but check if it's part of an acronym (U.S.) or a number (0.99).
It also splits correctly sentences after question or exclamation marks but not after commas.
Step6:  Most common tokens
What are the 20 most frequently occurring (unique) tokens in the text? What is their frequency?
The NTLK FreqDist class is used to encode “frequency distributions”, which count the number of times that something occurs, for example a token.
Its most_common() method then returns a list of tuples where each tuple is of the form (token, frequency). The list is sorted in descending order of frequency.
Step7:  Comma is the most common
Step8:  We can also remove any capital letters before tokenising
Step9:  Now we removed the punctuation and the capital letters but the most common token is "the", not a significative word ...
As we have seen last time, these are so-called stop words that are very common and are normally stripped from a text when doing these kind of analysis.
Meaningful most common tokens
A simple approach could be to filter the tokens that have a length greater than 5 and frequency of more than 150.
Step10:  This would work but would leave out also tokens such as I and you which are actually significative.
The better approach - that we have seen earlier how - is to remove stopwords using external files containing the stop words.
NLTK has a corpus of stop words in several languages
Step11:  Now we excluded words such as the but we can improve further the list by looking at semantically similar words, such as plural and singular versions.
Step12:  Stemming
Above, in the list of words we have both prince and princes which are respectively the singular and plural version of the same word (the stem). The same would happen with verb conjugation (love and loving are considered different words but are actually inflections of the same verb).
Stemmer is the tool that reduces such inflectional forms into their stem, base or root form and NLTK has several of them (each with a different heuristic algorithm).
Step13:  And now we apply one of the NLTK stemmer, the Porter stemmer
Step14:  As you see, all 5 different words have been reduced to the same stem and would be now the same lexical token.
Step15:  Now the word princ is counted 281 times, exactly like the sum of prince and princes.  
A note here
Step16:  Lemma
Lemmatization usually refers to doing things properly with the use of a vocabulary and morphological analysis of words, normally aiming to remove inflectional endings only and to return the base or dictionary form of a word, which is known as the lemma.
While a stemmer operates on a single word without knowledge of the context, a lemmatiser can take the context in consideration.  
NLTK has also a built-in lemmatiser, so let's see it in action
Step17:  We tell the lemmatise that the words are nouns. In this case it considers the same lemma words such as list (singular noun) and lists (plural noun) but leave as they are the other words.
Step18:  We get a different result if we say that the words are verbs.
They have all the same lemma, in fact they could be all different inflections or conjugation of a verb.
The type of words that can be used are
Step19:  It works with different adjectives, it doesn't look only at prefixes and suffixes.
You would wonder why stemmers are used, instead of always using lemmatisers
Step20:  Yes, the lemma now is prince.
But note that we consider all words in the book as nouns, while actually a proper way would be to apply the correct type to each single word.
Part of speech (PoS)
In traditional grammar, a part of speech (abbreviated form
Step21:  The NLTK function pos_tag() will tag each token with the estimated PoS.
NLTK has 13 categories of PoS. You can check the acronym using the NLTK help function
Step22:  Which are the most common PoS in The Prince book?
Step24:  It's not nouns (NN) but interections (IN) such as preposition or conjunction.
Extra note
<ASSISTANT_TASK:>
Python Code:
sampleText1 = "The Elephant's 4 legs: THE Pub! You can't believe it or can you, the believer?"
sampleText2 = "Pierre Vinken, 61 years old, will join the board as a nonexecutive director Nov. 29."
Explanation: Introduction to NLTK
We have seen how to do some basic text processing in Python, now we introduce an open source framework for natural language processing that can further help to work with human languages: NLTK (Natural Language ToolKit).
Tokenise a text
Let's start with a simple text in a Python string:
End of explanation
import nltk
s1Tokens = nltk.word_tokenize(sampleText1)
s1Tokens
len(s1Tokens)
Explanation: Tokens
The basic atomic part of each text are the tokens. A token is the NLP name for a sequence of characters that we want to treat as a group.
We have seen how we can extract tokens by splitting the text at the blank spaces.
NTLK has a function word_tokenize() for it:
End of explanation
s2Tokens = nltk.word_tokenize(sampleText2)
s2Tokens
Explanation: 21 tokens extracted, which include words and punctuation.
Note that the tokens are different than what a split by blank spaces would obtained, e.g. "can't" is by NTLK considered TWO tokens: "can" and "n't" (= "not") while a tokeniser that splits text by spaces would consider it a single token: "can't".
Let's see another example:
End of explanation
# If you would like to work with the raw text you can use 'bookRaw'
with open('../datasets/ThePrince.txt', 'r') as f:
    bookRaw = f.read()
bookTokens = nltk.word_tokenize(bookRaw)
bookText = nltk.Text(bookTokens) # special format
nBookTokens= len(bookTokens) # or alternatively len(bookText)
print ("*** Analysing book ***")    
print ("The book is {} chars long".format (len(bookRaw)))
print ("The book has {} tokens".format (nBookTokens))
Explanation: And we can apply it to an entire book, "The Prince" by Machiavelli that we used last time:
End of explanation
text1 = "This is the first sentence. A liter of milk in the U.S. costs $0.99. Is this the third sentence? Yes, it is!"
sentences = nltk.sent_tokenize(text1)
len(sentences)
sentences
Explanation: As mentioned above, the NTLK tokeniser works in a more sophisticated way than just splitting by spaces, therefore we got this time more tokens.
Sentences
NTLK has a function to tokenise a text not in words but in sentences.
End of explanation
sentences = nltk.sent_tokenize(bookRaw) # extract sentences
nSent = len(sentences)
print ("The book has {} sentences".format (nSent))
print ("and each sentence has in average {} tokens".format (nBookTokens / nSent))
Explanation: As you see, it is not splitting just after each full stop but check if it's part of an acronym (U.S.) or a number (0.99).
It also splits correctly sentences after question or exclamation marks but not after commas.
End of explanation
def get_top_words(tokens):
        # Calculate frequency distribution
    fdist = nltk.FreqDist(tokens)
    return fdist.most_common()
topBook = get_top_words(bookTokens)
  # Output top 20 words
topBook[:20]
Explanation: Most common tokens
What are the 20 most frequently occurring (unique) tokens in the text? What is their frequency?
The NTLK FreqDist class is used to encode “frequency distributions”, which count the number of times that something occurs, for example a token.
Its most_common() method then returns a list of tuples where each tuple is of the form (token, frequency). The list is sorted in descending order of frequency.
End of explanation
topWords = [(freq, word) for (word,freq) in topBook if word.isalpha() and freq > 400]
topWords
Explanation: Comma is the most common: we need to remove the punctuation.
Most common alphanumeric tokens
We can use isalpha() to check if the token is a word and not punctuation.
End of explanation
def preprocessText(text, lowercase=True):
    if lowercase:
        tokens = nltk.word_tokenize(text.lower())
    else:
        tokens = nltk.word_tokenize(text)
    return [word for word in tokens if word.isalpha()]
bookWords = preprocessText(bookRaw)
topBook = get_top_words(bookWords)
# Output top 20 words
topBook[:20]
print ("*** Analysing book ***")    
print ("The text has now {} words (tokens)".format (len(bookWords)))
Explanation: We can also remove any capital letters before tokenising:
End of explanation
meaningfulWords = [word for (word,freq) in topBook if len(word) > 5 and freq > 80]
sorted(meaningfulWords)
Explanation: Now we removed the punctuation and the capital letters but the most common token is "the", not a significative word ...
As we have seen last time, these are so-called stop words that are very common and are normally stripped from a text when doing these kind of analysis.
Meaningful most common tokens
A simple approach could be to filter the tokens that have a length greater than 5 and frequency of more than 150.
End of explanation
from nltk.corpus import stopwords
stopwordsEN = set(stopwords.words('english')) # english language
betterWords = [w for w in bookWords if w not in stopwordsEN]
topBook = get_top_words(betterWords)
# Output top 20 words
topBook[:20]
Explanation: This would work but would leave out also tokens such as I and you which are actually significative.
The better approach - that we have seen earlier how - is to remove stopwords using external files containing the stop words.
NLTK has a corpus of stop words in several languages:
End of explanation
'princes' in betterWords
betterWords.count("prince") + betterWords.count("princes") 
Explanation: Now we excluded words such as the but we can improve further the list by looking at semantically similar words, such as plural and singular versions.
End of explanation
input1 = "List listed lists listing listings"
words1 = input1.lower().split(' ')
words1
Explanation: Stemming
Above, in the list of words we have both prince and princes which are respectively the singular and plural version of the same word (the stem). The same would happen with verb conjugation (love and loving are considered different words but are actually inflections of the same verb).
Stemmer is the tool that reduces such inflectional forms into their stem, base or root form and NLTK has several of them (each with a different heuristic algorithm).
End of explanation
porter = nltk.PorterStemmer()
[porter.stem(t) for t in words1]
Explanation: And now we apply one of the NLTK stemmer, the Porter stemmer:
End of explanation
stemmedWords = [porter.stem(w) for w in betterWords]
topBook = get_top_words(stemmedWords)
topBook[:20]  # Output top 20 words
Explanation: As you see, all 5 different words have been reduced to the same stem and would be now the same lexical token.
End of explanation
from nltk.stem.snowball import SnowballStemmer
stemmerIT = SnowballStemmer("italian")
inputIT = "Io ho tre mele gialle, tu hai una mela gialla e due pere verdi"
wordsIT = inputIT.split(' ')
[stemmerIT.stem(w) for w in wordsIT]
Explanation: Now the word princ is counted 281 times, exactly like the sum of prince and princes.  
A note here: Stemming usually refers to a crude heuristic process that chops off the ends of words in the hope of achieving this goal correctly most of the time, and often includes the removal of derivational affixes.
Prince and princes become princ.
A different flavour is the lemmatisation that we will see in one second, but first a note about stemming in other languages than English.
Stemming in other languages
Snowball is an improvement created by Porter: a language to create stemmers and have rules for many more languages than English. 
For example Italian:
End of explanation
from nltk.stem import WordNetLemmatizer
lemmatizer = WordNetLemmatizer()
words1
[lemmatizer.lemmatize(w, 'n') for w in words1] # n = nouns
Explanation: Lemma
Lemmatization usually refers to doing things properly with the use of a vocabulary and morphological analysis of words, normally aiming to remove inflectional endings only and to return the base or dictionary form of a word, which is known as the lemma.
While a stemmer operates on a single word without knowledge of the context, a lemmatiser can take the context in consideration.  
NLTK has also a built-in lemmatiser, so let's see it in action:
End of explanation
[lemmatizer.lemmatize(w, 'v') for w in words1] # v = verbs
Explanation: We tell the lemmatise that the words are nouns. In this case it considers the same lemma words such as list (singular noun) and lists (plural noun) but leave as they are the other words.
End of explanation
words2 = ['good', 'better']
[porter.stem(w) for w in words2]
[lemmatizer.lemmatize(w, 'a') for w in words2] 
Explanation: We get a different result if we say that the words are verbs.
They have all the same lemma, in fact they could be all different inflections or conjugation of a verb.
The type of words that can be used are:
'n' = noun, 'v'=verb, 'a'=adjective, 'r'=adverb
End of explanation
lemmatisedWords = [lemmatizer.lemmatize(w, 'n') for w in betterWords]
topBook = get_top_words(lemmatisedWords)
topBook[:20]  # Output top 20 words
Explanation: It works with different adjectives, it doesn't look only at prefixes and suffixes.
You would wonder why stemmers are used, instead of always using lemmatisers: stemmers are much simpler, smaller and faster and for many applications good enough.
Now we lemmatise the book:
End of explanation
text1 = "Children shouldn't drink a sugary drink before bed."
tokensT1 = nltk.word_tokenize(text1)
nltk.pos_tag(tokensT1)
Explanation: Yes, the lemma now is prince.
But note that we consider all words in the book as nouns, while actually a proper way would be to apply the correct type to each single word.
Part of speech (PoS)
In traditional grammar, a part of speech (abbreviated form: PoS or POS) is a category of words which have similar grammatical properties. 
For example, an adjective (red, big, quiet, ...) describe properties while a verb (throw, walk, have) describe actions or states.
Commonly listed parts of speech are noun, verb, adjective, adverb, pronoun, preposition, conjunction, interjection.
End of explanation
nltk.help.upenn_tagset('RB')
Explanation: The NLTK function pos_tag() will tag each token with the estimated PoS.
NLTK has 13 categories of PoS. You can check the acronym using the NLTK help function:
End of explanation
tokensAndPos = nltk.pos_tag(bookTokens)
posList = [thePOS for (word, thePOS) in tokensAndPos]
fdistPos = nltk.FreqDist(posList)
fdistPos.most_common(5)
nltk.help.upenn_tagset('IN')
Explanation: Which are the most common PoS in The Prince book?
End of explanation
# Parsing sentence structure
text2 = nltk.word_tokenize("Alice loves Bob")
grammar = nltk.CFG.fromstring(
S -> NP VP
VP -> V NP
NP -> 'Alice' | 'Bob'
V -> 'loves'
)
parser = nltk.ChartParser(grammar)
trees = parser.parse_all(text2)
for tree in trees:
    print(tree)
Explanation: It's not nouns (NN) but interections (IN) such as preposition or conjunction.
Extra note: Parsing the grammar structure
Words can be ambiguous and sometimes is not easy to understand which kind of POS is a word, for example in the sentence "visiting aunts can be a nuisance", is visiting a verb or an adjective?
Tagging a PoS depends on the context, which can be ambiguous.
Making sense of a sentence is easier if it follows a well-defined grammatical structure, such as : subject + verb + object
NLTK allows to define a formal grammar which can then be used to parse a text.  The NLTK ChartParser is a procedure for finding one or more trees (sentences have internal organisation that can be represented using a tree) corresponding to a grammatically well-formed sentence.
End of explanation
<END_TASK>