<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Tokenisation of Text into sentences
Code from Mastering Natural Langiage Processing with Python.
Step1:  For large number of senteces try this instead
Step2:  For splitting sentences into words
This can be done using the word_tokenize as per below
Step3:  What about other tokenizers
There is the TreeBankTokenizer which tokenises according to Penn Treebank Corpus conventions.
Step4:  What about using regular expressions
Regular expressions can be used to split text based on punctuation chars. Luckily, we won't need RegExs just yet
Step5:  But if you want to use regular expressions
Step6:  Instead of instantiating a class, here is an alternative way to tokenize with regexs
Step7:  And here is how we can
Step8:  How about selecting words with a capital letter
Step9:  Other RegeEx tokenizers
And what about using predefined regexs?
Step10:  Tokenisation using a white_space tokenizer
Step11:  The split method can also be used to specify a white_space character or anything else
Step12:  The SpaceTokenizer works in a very similar way to sent.split(' ')
Step13:  What if all we want it to tokenize words into lines
Step14:  Normalisation
It is the process of
Step15:  Let's convert text to lowercase and uppercase
Step16:  Stopwords
Also, another important part is stopwords. Stopwords are words that need to be filtered out as they appear way too often and do not provide too much useful information
Step17:  Substituting and correcting tokens
Step19:  Delete repeating characters
Have a look at this
Step21:  Replacing words witht heir synonyms
Step22:  Zipfs Law
It states that the frequency of a token in a text is directly proportional to its rank or position in the sorted list.
Step23:  Similarity Measures
Many similarity measures can be used to quantify the similarity between words, phrases or bigger chanks of text. The nltk.metrics lib is used for this task
Step24:  Edit distance
Minimum number of insertions, deletions or substitutions necessary to convert one word into another.
Step25:  Jaccard Coefficient
A measure of the overlap between two sets.
Step26:  Other Similarity algorithms
The binary distance which returns 0 if the two strings are exactly the same and 1 if they are not.
Step27:  Or the masi distance, which is based on partial agreement when multiple labels are present
<ASSISTANT_TASK:>
Python Code:
import nltk
# nltk.download()
text="Welcome readers. I hope you find it interesting. please fo reply."
from nltk.tokenize import sent_tokenize
sent_tokenize(text)
Explanation: Tokenisation of Text into sentences
Code from Mastering Natural Langiage Processing with Python.
End of explanation
tokenizer=nltk.data.load('tokenizers/punkt/english.pickle')
text='Hello everyone. Hope you are all fine and doing well. Hope that you will find the book interesting.'
tokenizer.tokenize(text)
Explanation: For large number of senteces try this instead
End of explanation
text=nltk.word_tokenize("PeirreVinken, 59 years old, will join as a nonexecutive director on Nov. 29.")
print(text)
r=input("Please provide some imput text:")
from nltk import word_tokenize
print("The length of this word is", len(word_tokenize(r)),"words.")
Explanation: For splitting sentences into words
This can be done using the word_tokenize as per below:
End of explanation
from nltk.tokenize import TreebankWordTokenizer
tokenizer = TreebankWordTokenizer()
print(tokenizer.tokenize("Have a nice day. I hope you find the book interesting"))
text=nltk.word_tokenize(" Don't hesitate to ask questions")
print(text)
Explanation: What about other tokenizers
There is the TreeBankTokenizer which tokenises according to Penn Treebank Corpus conventions.
End of explanation
from nltk.tokenize import WordPunctTokenizer
tokenizer=WordPunctTokenizer()
print(tokenizer.tokenize(" Don't hesitate to ask questions"))
Explanation: What about using regular expressions
Regular expressions can be used to split text based on punctuation chars. Luckily, we won't need RegExs just yet:
End of explanation
from nltk.tokenize import RegexpTokenizer
tokenizer=RegexpTokenizer("[\w']+")
print(tokenizer.tokenize("Don't hesitate to ask questions"))
Explanation: But if you want to use regular expressions:
End of explanation
from nltk.tokenize import regexp_tokenize
sent="Don't hesitate to ask questions"
print(regexp_tokenize(sent, pattern='\w+|\$[\d\.]+|\S+'))
Explanation: Instead of instantiating a class, here is an alternative way to tokenize with regexs:
End of explanation
tokenizer=RegexpTokenizer('[\'\w\-]+',gaps=False)
print(tokenizer.tokenize("Don't hesitate to ask questions. Hi there! Antoio-b was 54 years old"))
Explanation: And here is how we can
End of explanation
sent=" She secured 90.56 % in class X . She is a meritorious student"
capt = RegexpTokenizer('[A-Z]\w+')
print(capt.tokenize(sent))
Explanation: How about selecting words with a capital letter:
End of explanation
sent=" She secured 90.56 % in class X . She is a meritorious student"
from nltk.tokenize import BlanklineTokenizer
print(BlanklineTokenizer().tokenize(sent))
Explanation: Other RegeEx tokenizers
And what about using predefined regexs?
End of explanation
sent=" She secured 90.56 % in class X . She is a meritorious student"
from nltk.tokenize import WhitespaceTokenizer
print(WhitespaceTokenizer().tokenize(sent))
Explanation: Tokenisation using a white_space tokenizer:
End of explanation
sent= "She secured 90.56 % in class X. She is a meritorious student"
print(sent.split())     # Notice that these two...
print(sent.split(' '))  # ...are equivalent
sent=" She secured 90.56 % in class X \n. She is a meritorious student\n"
print(sent.split('\n'))
Explanation: The split method can also be used to specify a white_space character or anything else:
End of explanation
sent=" She secured 90.56 % in class X \n. She is a meritorious student\n"
from nltk.tokenize import SpaceTokenizer
print(SpaceTokenizer().tokenize(sent))
Explanation: The SpaceTokenizer works in a very similar way to sent.split(' ')
End of explanation
from nltk.tokenize import BlanklineTokenizer
sent=" She secured 90.56 % in class X \n. She is a meritorious student\n"
print(BlanklineTokenizer().tokenize(sent))
from nltk.tokenize import LineTokenizer
print(LineTokenizer(blanklines='keep').tokenize(sent))
print(LineTokenizer(blanklines='discard').tokenize(sent))
Explanation: What if all we want it to tokenize words into lines:
End of explanation
import re
import string
text=[" It is a pleasant evening.","Guests, who came from US arrived at the venue","Food was tasty."]
from nltk.tokenize import word_tokenize
tokenized_docs=[word_tokenize(doc) for doc in text]
x=re.compile('[%s]' % re.escape(string.punctuation))
tokenized_docs_no_punctuation = []
for review in tokenized_docs:
    new_review = []
    for token in review: 
        new_token = x.sub(u'', token)
        if not new_token == u'':
            new_review.append(new_token)
    tokenized_docs_no_punctuation.append(new_review)	
print(tokenized_docs_no_punctuation)
Explanation: Normalisation
It is the process of: 
1. eliminating punctuation, 
2. converting the entire text into lowercase or uppercase, 
3. changing numbers into words, 
4. expanding abbreviations, 
5. canonicalisation of text 
and so on.
Let's start with punctuation:
End of explanation
text='HARdWork IS KEy to SUCCESS'
print(text.lower())
print(text.upper())
Explanation: Let's convert text to lowercase and uppercase:
End of explanation
from nltk.corpus import stopwords
stops=set(stopwords.words('english'))
words=["Don't", 'hesitate','to','ask','questions']
print([word for word in words if word not in stops])
# For other languages:
print(stopwords.fileids())
# Lets calculate stopwords in english
print(stopwords.words('english'))
def para_fraction(text):
    stopwords = nltk.corpus.stopwords.words('english')
    para = [w for w in text if w.lower() not in stopwords]
    return len(para) / len(text)
print(para_fraction(nltk.corpus.reuters.words()))
print(para_fraction(nltk.corpus.inaugural.words()))
Explanation: Stopwords
Also, another important part is stopwords. Stopwords are words that need to be filtered out as they appear way too often and do not provide too much useful information:
End of explanation
# Using regular expressions
import re
replacement_patterns = [
(r'won\'t', 'will not'),
(r'can\'t', 'cannot'),
(r'i\'m', 'i am'),
(r'ain\'t', 'is not'),
(r'(\w+)\'ll', '\g<1> will'),
(r'(\w+)n\'t', '\g<1> not'),
(r'(\w+)\'ve', '\g<1> have'),
(r'(\w+)\'s', '\g<1> is'),
(r'(\w+)\'re', '\g<1> are'),
(r'(\w+)\'d', '\g<1> would')
 ]
class RegexpReplacer(object):
   def __init__(self, patterns=replacement_patterns):
      # Fixed this line - "patterns", not "pattern"
      self.patterns = [(re.compile(regex), repl) for (regex, repl) in patterns]
   def replace(self, text):
      s = text
      for (pattern, repl) in self.patterns:
          (s, count) = re.subn(pattern, repl, s)
      # Fixed indentation here
      return s
rep=RegexpReplacer()
print(rep.replace("can't is a contradicton"))
replacer= RegexpReplacer()
replacer.replace("Don't hesitate to ask questions")
print(replacer.replace("She must've gone to the market but she didn't go"))
# Substitution can take place before tokenisation
replacer=RegexpReplacer()
word_tokenize("Don't hesitate to ask questions")
print(word_tokenize(replacer.replace("Don't hesitate to ask questions")))
Explanation: Substituting and correcting tokens
End of explanation
from nltk.corpus import wordnet
class RepeatReplacer(object):
	 Removes repeating characters until a valid word is found.
	>>> replacer = RepeatReplacer()
	>>> replacer.replace('looooove')
	'love'
	>>> replacer.replace('oooooh')
	'ooh'
	>>> replacer.replace('goose')
	'goose'
	
	def __init__(self):
		self.repeat_regexp = re.compile(r'(\w*)(\w)\2(\w*)')
		self.repl = r'\1\2\3'
	def replace(self, word):
		if wordnet.synsets(word):
			return word
		
		repl_word = self.repeat_regexp.sub(self.repl, word)
		
		if repl_word != word:
			return self.replace(repl_word)
		else:
			return repl_word
replacer=RepeatReplacer()
print(replacer.replace('lotttt'))
print(replacer.replace('ohhhhh'))
print(replacer.replace('ooohhhhh'))
Explanation: Delete repeating characters
Have a look at this
End of explanation
class WordReplacer(object):
	 WordReplacer that replaces a given word with a word from the word_map,
	or if the word isn't found, returns the word as is.
	>>> replacer = WordReplacer({'bday': 'birthday'})
	>>> replacer.replace('bday')
	'birthday'
	>>> replacer.replace('happy')
	'happy'
	
	def __init__(self, word_map):
		self.word_map = word_map
	
	def replace(self, word):
		return self.word_map.get(word, word)
replacer=WordReplacer({'congrats':'congratulations'})
print(replacer.replace('congrats'))
print(replacer.replace('maths'))
Explanation: Replacing words witht heir synonyms
End of explanation
from nltk.corpus import gutenberg
from nltk.probability import FreqDist
import matplotlib as mpl
import matplotlib.pyplot as plt
%matplotlib inline
fd = FreqDist()
for text in gutenberg.fileids():
    for word in gutenberg.words(text):
        fd[word] += 1
ranks = []
freqs = []
for rank, word in enumerate(fd):
    ranks.append(rank+1)
    freqs.append(fd[word])
plt.loglog(ranks,freqs)
plt.xlabel('frequency(f)', fontsize=14, fontweight='bold')
plt.ylabel('rank(r)', fontsize=14, fontweight='bold')
plt.grid(True)
fig = plt.figure(figsize=(40,20), dpi= 80, facecolor='w', edgecolor='k')
fig
Explanation: Zipfs Law
It states that the frequency of a token in a text is directly proportional to its rank or position in the sorted list.
End of explanation
from __future__ import print_function
from nltk.metrics import *
training='PERSON OTHER PERSON OTHER OTHER ORGANIZATION'.split()
testing='PERSON OTHER OTHER OTHER OTHER OTHER'.split()
print('Accuracy:' + str(accuracy(training,testing)))
trainset=set(training)
testset=set(testing)
precision(trainset,testset)
print('Recall:' + str(recall(trainset,testset)))
print('F_measure:' + str(f_measure(trainset,testset)))
Explanation: Similarity Measures
Many similarity measures can be used to quantify the similarity between words, phrases or bigger chanks of text. The nltk.metrics lib is used for this task:
End of explanation
from nltk.metrics import *
print(edit_distance("relate","relation"))
print(edit_distance("suggestion","calculation"))
Explanation: Edit distance
Minimum number of insertions, deletions or substitutions necessary to convert one word into another.
End of explanation
X=set([10,20,30,40])
Y=set([20,30,60])
print(jaccard_distance(X,Y))
Explanation: Jaccard Coefficient
A measure of the overlap between two sets.
End of explanation
X = set([10,20,30,40])
Y= set([30,50,70])
print(binary_distance(X, Y))
Explanation: Other Similarity algorithms
The binary distance which returns 0 if the two strings are exactly the same and 1 if they are not.
End of explanation
X=set([10,20,30,40])
Y=set([30,50,70])
print(masi_distance(X,Y))
Explanation: Or the masi distance, which is based on partial agreement when multiple labels are present:
End of explanation
<END_TASK>