Use of PunktSentenceTokenizer in NLTK
<p>I am learning Natural Language Processing using NLTK.
I came across the code using <code>PunktSentenceTokenizer</code> whose actual use I cannot understand in the given code. The code is given :</p>

<pre><code>import nltk
from nltk.corpus import state_union
from nltk.tokenize import PunktSentenceTokenizer

train_text = state_union.raw("2005-GWBush.txt")
sample_text = state_union.raw("2006-GWBush.txt")

custom_sent_tokenizer = PunktSentenceTokenizer(train_text) #A

tokenized = custom_sent_tokenizer.tokenize(sample_text)   #B

def process_content():
try:
    for i in tokenized[:5]:
        words = nltk.word_tokenize(i)
        tagged = nltk.pos_tag(words)
        print(tagged)

except Exception as e:
    print(str(e))


process_content()
</code></pre>

<p>So, why do we use PunktSentenceTokenizer.  And what is going on in the line marked A and B. I mean there is a training text and the other a sample text, but what is the need for two data sets to get the Part of Speech tagging.</p>

<p>Line marked as <code>A</code> and <code>B</code> is which I am not able to understand.</p>

<p>PS : I did try to look in the NLTK book but could not understand what is the real use of PunktSentenceTokenizer</p>