<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 NLTK experiments
based on NLTK with Python 3 for Natural Language Processing by Sentdex
Step1:  Tokenizing
based on 
- https
Step2:  Stop words
sources
video
Step3:  Stemming
source
video
Step4:  Part of Speech Tagging
source
video
```
POS tag list
Step6:  Chunking
source
video
<ASSISTANT_TASK:>
Python Code:
import nltk
from nltk import tokenize
# TODO: we don't relly want to download packages each time when we lauch this script
# so it'll better to check somehow whether we have packages or not - or Download on demand
# nltk.download()
Explanation: NLTK experiments
based on NLTK with Python 3 for Natural Language Processing by Sentdex
End of explanation
example = 'Hello Mr. Smith, how are you doing today? The weather is great, and Python is awesome. ' \
          'The sky is pinkish-blue. You shouldn\'t eat cardboard.'
tokenize.sent_tokenize(example)
tokenize.word_tokenize(example)
Explanation: Tokenizing
based on 
- https://pythonprogramming.net/tokenizing-words-sentences-nltk-tutorial/
- youtube
End of explanation
from nltk import corpus, tokenize
example_sentence = 'This is a sample sentence, showing off the stop words filtration.'
stop_words = set(corpus.stopwords.words('english'))
words = tokenize.word_tokenize(example_sentence)
filtered_sentence = [w for w in words if w not in stop_words]
print(filtered_sentence)
Explanation: Stop words
sources
video
End of explanation
from nltk import stem, tokenize
ps = stem.PorterStemmer()
example_words = ['python', 'pythoner', 'pythoning', 'pythoned', 'pythonly', 'pythonic', 'pythonista']
['{} --> {}'.format(w, ps.stem(w)) for w in example_words]
example_text = 'It is important to by very pythonly while you are pythoning with python. '\
               'All pythoners have pythoned poorly at least once.'
['{} --> {}'.format(w, ps.stem(w)) for w in tokenize.word_tokenize(example_text)]
Explanation: Stemming
source
video
End of explanation
import nltk
from nltk import corpus, tokenize
train_text = corpus.state_union.raw('2005-GWBush.txt')
sample_text = corpus.state_union.raw('2006-GWBush.txt')
# Map tag to description, useful for annotations
tag_to_description = {
    'CC': 'coordinating conjunction',
    'CD': 'cardinal digit',
    'DT': 'determiner',
    'EX': 'existential there (like: "there is" ... think of it like "there exists")',
    'FW': 'foreign word',
    'IN': 'preposition/subordinating conjunction',
    'JJ': 'adjective	"big"',
    'JJR': 'adjective, comparative	"bigger"',
    'JJS': 'adjective, superlative	"biggest"',
    'LS': 'list marker	1)',
    'MD': 'modal	could, will',
    'NN': 'noun, singular "desk"',
    'NNS': 'noun plural	"desks"',
    'NNP': 'proper noun, singular	"Harrison"',
    'NNPS': 'proper noun, plural	"Americans"',
    'PDT': 'predeterminer	"all tdhe kids"',
    'POS': 'possessive ending	parent"s',
    'PRP': 'personal pronoundß	I, he, she',
    'PRP$': 'possessive pronoun	my, his, hers',
    'RB': 'adverb	very, silently,',
    'RBR': 'adverb, comparative	better',
    'RBS': 'adverb, superlative	best',
    'RP': 'particle	give up',
    'TO': 'to	go "to" the store.',
    'UH': 'interjection	errrrrrrrm',
    'VB': 'verb, base form	take',
    'VBD': 'verb, past tense	took',
    'VBG': 'verb, gerund/present participle	taking',
    'VBN': 'verb, past participle	taken',
    'VBP': 'verb, sing. present, non-3d	take',
    'VBZ': 'verb, 3rd person sing. present	takes',
    'WDT': 'wh-determiner	which',
    'WP': 'wh-pronoun	who, what',
    'WP$': 'possessive wh-pronoun	whose',
    'WRB': 'wh-abverb	where, when',
}
from collections import Counter
from operator import itemgetter, attrgetter
custom_sent_tokenizer = tokenize.PunktSentenceTokenizer(train_text)
tokenized_text = custom_sent_tokenizer.tokenize(sample_text)
total_counts = Counter()
for i in tokenized_text[:5]:
    words = nltk.word_tokenize(i)
    tagged = nltk.pos_tag(words)
    print('# Sentence:')
    print(i)
    print('# Words:')
    print(words)
    print('# Tagged:')
    print(tagged)
    counts = Counter(tag for word, tag in tagged)
    total_counts += counts
    print('\n')
total = sum(total_counts.values())
freq = dict((word, float(count) / total) for word, count in sorted(total_counts.items()))
print('# Counts:')
print('\n\n-----\n\n'.join(['{}\n[{}] {}'.format(f, tag, tag_to_description.get(tag, tag)) for tag, f in sorted(freq.items(), key=itemgetter(1), reverse=True)]))
Explanation: Part of Speech Tagging
source
video
```
POS tag list:
CC  coordinating conjunction
CD  cardinal digit
DT  determiner
EX  existential there (like: "there is" ... think of it like "there exists")
FW  foreign word
IN  preposition/subordinating conjunction
JJ  adjective   'big'
JJR adjective, comparative  'bigger'
JJS adjective, superlative  'biggest'
LS  list marker 1)
MD  modal   could, will
NN  noun, singular 'desk'
NNS noun plural 'desks'
NNP proper noun, singular   'Harrison'
NNPS    proper noun, plural 'Americans'
PDT predeterminer   'all tdhe kids'
POS possessive ending   parent's
PRP personal pronoundß  I, he, she
PRP$    possessive pronoun  my, his, hers
RB  adverb  very, silently,
RBR adverb, comparative better
RBS adverb, superlative best
RP  particle    give up
TO  to  go 'to' the store.
UH  interjection    errrrrrrrm
VB  verb, base form take
VBD verb, past tense    took
VBG verb, gerund/present participle taking
VBN verb, past participle   taken
VBP verb, sing. present, non-3d take
VBZ verb, 3rd person sing. present  takes
WDT wh-determiner   which
WP  wh-pronoun  who, what
WP$ possessive wh-pronoun   whose
WRB wh-abverb   where, when
```
End of explanation
%matplotlib inline
import matplotlib as mpl
import matplotlib.pyplot as plt
chunkGram = rChunk: {<RB.?>*<VB.?>*<NNP>+<NN>?}
chunkParser = nltk.RegexpParser(chunkGram)
for i in tokenized_text[:5]:
    words = nltk.word_tokenize(i)
    tagged = nltk.pos_tag(words)
    chunked = chunkParser.parse(tagged)
    # TODO: should fix it
    # I'm using jupyter inside of Docker so maybe it is reason why doesn't work :(
    # I've found this one https://stackoverflow.com/questions/31779707/how-do-you-make-nltk-draw-trees-that-are-inline-in-ipython-jupyter
    # but haven't checkit it yet.
    chunked.draw()
Explanation: Chunking
source
video
End of explanation
<END_TASK>