Preventing splitting at apostrophies when tokenizing words using nltk
<p>I am using the <code>nltk</code> to split up sentences to words. e.g. </p>

<pre><code> nltk.word_tokenize("The code didn't work!")
 -&gt; ['The', 'code', 'did', "n't", 'work', '!']
</code></pre>

<p>The tokenizing works well at spliting up word boundaries [i.e. splitting punctuation from words], but sometimes over-splits, and modifiers at the end of the word get treated as separate parts. For example, <code>didn't</code> gets split into the parts <code>did</code> and <code>n't</code> and <code>i've</code> gets split to <code>I</code> and <code>'ve</code>. Obviously this is because such words are split in two in the original corpus that <code>nltk</code> is using, and may be desirable in some instances.</p>

<p>Is there any built in way of over-riding this behavior? Possibly in a similar manner to how <code>nltk's</code> <code>MWETokenizer</code> is able to aggregate multiple words to phrases, but in this case to just aggregate word components to words.</p>

<p>Alternatively, is there another tokenizer that does not split up word-parts?</p>