<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 <h1>Natural Language Preprocessing</h1>
<br>
<em><b>Gregory Antell & Emily Halket</b></em>
<br>
<em><b>December, 2016</b></em>
This notebook provides a brief overview of common steps taken natural language preprocessing. The goal is to get you started thinking about how to process your data, not to provide a formal pipeline. (add another few background sentences here)
<p>Preprocessing follows a general series of steps, each requiring decisions that can substantially impact the final output if not considered carefully. For this tutorial, we will be emphasizing how different sources of text require different approaches for preprocessing and modeling. As you approach your own data, think about the implications of each decision on the outcome of your analysis.</p>
<h2>Requirements</h2>
<p>This tutorial requires several commonly used Python packages for data analysis and Natural Language Processing (NLP)
Step1:  <h2>Data</h2>
<p>Here we will be exploring two different data sets
Step2:  Let's take a peek at the raw text of this article to see what we are dealing with!
Right off the bat you can see that we have a mixture of uppercase and lowercase words, punctuation, and some character encoding.
Step3:  <h2>Preprocessing Text</h2>
<p> After looking at our raw text, we know that there are a number of textual attributes that we will need to address before we can ultimately represent our text as quantified features. Using some built in string functions, we can address the character encoding and mixed capitalization.
Step4:  <h3>1. Tokenization</h3>
<p>In order to process text, it must be deconstructed into its constituent elements through a process termed <b><em>tokenization</em></b>. Often, the <b><em>tokens</em></b> yielded from this process are individual words in a document. Tokens represent the linguistic units of a document.</p>
<p>A simplistic way to tokenize text relies on white space, such as in <code>nltk.tokenize.WhitespaceTokenizer</code>. Relying on white space, however, does not take <b>punctuation</b> into account, and depending on this some tokens will include punctuation  and will require further preprocessing (e.g. 'account,'). Depending on your data, the punctuation may provide meaningful information, so you will want to think about whether it should be preserved or if it can be removed. Tokenization is particularly challenging in the biomedical field, where many phrases contain substantial punctuation (parentheses, hyphens, etc.) and negation detection is critical.</p>
<p>NLTK contains many built-in modules for tokenization, such as <code>nltk.tokenize.WhitespaceTokenizer</code> and <code>nltk.tokenize.RegexpTokenizer</code>.
<p>See also
Step5:  Example
Step6:  <h3>2. Stop Words</h3>
<p>Depending on the application, many words provide little value when building an NLP model. Accordingly, these are termed <b><em>stop words</em></b>. Examples of stop words include pronouns, articles, prepositions and conjunctions, but there are many other words, or non meaningful tokens, that you may wish to remove. For instance, there may be artifacts from the web scraping process that you need to remove. </p>
<p>Stop words can be determined and handled in many different ways, including
Step7:  Let's remove the stop words and compare to our original list of tokens from our regular expression tokenizer.
Step8:  You can see that by removing stop words, we now have less than half the number of tokens as our original list. Taking a peek at the cleaned tokens, we can see that a lot of the information that makes the sentence read like something a human would expect has been lost but the key nouns, verbs, adjectives, and adverbs remain.
Step9:  You may notice from looking at this sample, however, that a potentially meaningful word has been removed
Step10:  While <b><em>stemming</em></b> is a heuristic process that selectively removes the end of words, <b><em>lemmatization</em></b> is a more sophisticated process that takes into account variables such as part-of-speech, meaning, and context within a document or neighboring sentences.</p>
Step11:  <p>In this example, lemmatization retains a bit more information than stemming. Within stemming, the Lancaster method is more aggressive than Porter and Snowball. Remember that this step allows us to reduce words to a common base form so that we can reduce our feature space and perform counting of occurrences. It will depend on your data and your application as to how much information you need to retain. </p>
<p>See also
Step12:  Let's take a look at a sample of our stemmed tokens
Step13:  In contrast, here are the same tokens in their lemmatized form
Step14:  <h3>4. Vectorization </h3>
<p> Often in natural language processing we want to represent our text as a quantitative set of features for subsequent analysis. One way to generate features from text is to count the occurrences words. This apporoach is often referred to as a bag of words approach.</p>
<p>In the example of our article, we could represent the article as a vector of counts for each token. If we did the same for all of the other articles, we would have a set of vectors with each vector representing an article. If we had only one article, then we could have split the article into sentences and then represented each sentence as a vector. </p>
<p>If we apply a count vectorizer to our article, we will have a vector with the length of the number of unique tokens. </p>
Example
<ASSISTANT_TASK:>
Python Code:
# import requirements
import pandas as pd
import nltk
import gensim
import spacy
Explanation: <h1>Natural Language Preprocessing</h1>
<br>
<em><b>Gregory Antell & Emily Halket</b></em>
<br>
<em><b>December, 2016</b></em>
This notebook provides a brief overview of common steps taken natural language preprocessing. The goal is to get you started thinking about how to process your data, not to provide a formal pipeline. (add another few background sentences here)
<p>Preprocessing follows a general series of steps, each requiring decisions that can substantially impact the final output if not considered carefully. For this tutorial, we will be emphasizing how different sources of text require different approaches for preprocessing and modeling. As you approach your own data, think about the implications of each decision on the outcome of your analysis.</p>
<h2>Requirements</h2>
<p>This tutorial requires several commonly used Python packages for data analysis and Natural Language Processing (NLP):</p>
<ul>
<li><b>Pandas: </b>for data structures and analysis in Python
<li><b>NLTK: </b>Natural Language Toolkit
<li><b>gensim: </b>for topic modelling
</ul>
End of explanation
# read subset of data from csv file into panadas dataframe
df = pd.read_csv('1_100.csv')
# for now, chosing one article to illustrate preprocessing
article = df['full_text'][939]
Explanation: <h2>Data</h2>
<p>Here we will be exploring two different data sets:</p>
<ol>
<li>New York Times op-eds
<li>Stack Overflow questions and comments
</ol>
<p>While the New York Times data set consists of traditional English prose and substantially longer articles, the Stack Overflow data set is vastly different. It contains <b> Finish statement later? Also, this part may want to be moved to a second section where we actually do the comparison </b></p>
<p>In this repository, there is a subset of 100 op-ed articles from the New York Times. We will read these articles into a data frame. We will start off by looking at one article to illustrate the steps of preprocessing, and then we will compare both data sets to illustrate how the process is informed by the nature of the data. </p>
End of explanation
article[:500]
Explanation: Let's take a peek at the raw text of this article to see what we are dealing with!
Right off the bat you can see that we have a mixture of uppercase and lowercase words, punctuation, and some character encoding.
End of explanation
article[:500].decode('utf-8').lower()
Explanation: <h2>Preprocessing Text</h2>
<p> After looking at our raw text, we know that there are a number of textual attributes that we will need to address before we can ultimately represent our text as quantified features. Using some built in string functions, we can address the character encoding and mixed capitalization.
End of explanation
from nltk.tokenize import WhitespaceTokenizer
ws_tokenizer = WhitespaceTokenizer()
# tokenize example document
nyt_ws_tokens = ws_tokenizer.tokenize(article.decode('utf-8').lower())
print nyt_ws_tokens[:75]
Explanation: <h3>1. Tokenization</h3>
<p>In order to process text, it must be deconstructed into its constituent elements through a process termed <b><em>tokenization</em></b>. Often, the <b><em>tokens</em></b> yielded from this process are individual words in a document. Tokens represent the linguistic units of a document.</p>
<p>A simplistic way to tokenize text relies on white space, such as in <code>nltk.tokenize.WhitespaceTokenizer</code>. Relying on white space, however, does not take <b>punctuation</b> into account, and depending on this some tokens will include punctuation  and will require further preprocessing (e.g. 'account,'). Depending on your data, the punctuation may provide meaningful information, so you will want to think about whether it should be preserved or if it can be removed. Tokenization is particularly challenging in the biomedical field, where many phrases contain substantial punctuation (parentheses, hyphens, etc.) and negation detection is critical.</p>
<p>NLTK contains many built-in modules for tokenization, such as <code>nltk.tokenize.WhitespaceTokenizer</code> and <code>nltk.tokenize.RegexpTokenizer</code>.
<p>See also:
<br>
<a href=https://www.ibm.com/developerworks/community/blogs/nlp/entry/tokenization?lang=en>The Art of Tokenization</a></p>
<a href=https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4231086/>Negation's Not Solved: Generalizability Versus Optimizability in Clinical Natural Language Processing</a></p>
Example: Whitespace Tokenization
Here we apply the Whitespace Tokenizer on the sample article. Notice that we are again decoding characters (such as quotation marks) and using all lowercase characters. Because we used white space as the marker between tokens, we still have punctuation (e.g. 'life.' and '\u201cif')
End of explanation
from nltk.tokenize import RegexpTokenizer
re_tokenizer = RegexpTokenizer(r'\w+')
nyt_re_tokens = re_tokenizer.tokenize(article.decode('utf-8').lower())
print nyt_re_tokens[:100]
Explanation: Example: Regular Expression Tokenization
By applying the regular expression tokenizer we can return a list of word tokens without punctuation.
End of explanation
from nltk.corpus import stopwords
# print the first 5 standard English stop words
stop_list = [w for w in stopwords.words('english')]
print stop_list[:5]
# print the type of the elements in the stop words list
print type(stop_list[0])
Explanation: <h3>2. Stop Words</h3>
<p>Depending on the application, many words provide little value when building an NLP model. Accordingly, these are termed <b><em>stop words</em></b>. Examples of stop words include pronouns, articles, prepositions and conjunctions, but there are many other words, or non meaningful tokens, that you may wish to remove. For instance, there may be artifacts from the web scraping process that you need to remove. </p>
<p>Stop words can be determined and handled in many different ways, including:
<ul>
<li>Using a list of words determined <em>a priori</em>, either a standard list from the NLTK package or one modified from such a list based on domain knowledge of a particular subject
<br><br>
<li>Sorting the terms by <b><em>collection frequency</em></b> (the total number of times each term appears in the document collection), and then to taking the most frequent terms as a stop list based on semantic content.
<br><br>
<li>Using no defined stop list at all, and dealing with text data in a purely statistical manner. In general, search engines do not use stop lists.
</ul>
As you work with your text, you may decide to iterate on this process. See also: <a href=http://nlp.stanford.edu/IR-book/html/htmledition/dropping-common-terms-stop-words-1.html>Stop Words</a>
#### Example: Stopword Corpus
For this example, we will use the english stopword corpus from NLTK.
End of explanation
cleaned_tokens = []
stop_words = set(stopwords.words('english'))
for token in nyt_re_tokens:
    if token not in stop_words:
        cleaned_tokens.append(token)
print 'Number of tokens before removing stop words: %d' % len(nyt_re_tokens)
print 'Number of tokens after removing stop words: %d' % len(cleaned_tokens)
Explanation: Let's remove the stop words and compare to our original list of tokens from our regular expression tokenizer.
End of explanation
print cleaned_tokens[:50]
Explanation: You can see that by removing stop words, we now have less than half the number of tokens as our original list. Taking a peek at the cleaned tokens, we can see that a lot of the information that makes the sentence read like something a human would expect has been lost but the key nouns, verbs, adjectives, and adverbs remain.
End of explanation
from nltk.stem.porter import PorterStemmer
from nltk.stem.snowball import SnowballStemmer
from nltk.stem.lancaster import LancasterStemmer
porter = PorterStemmer()
snowball = SnowballStemmer('english')
lancaster = LancasterStemmer()
print 'Porter Stem of "explanation": %s' % porter.stem('explanation')
print 'Porter2 (Snowball) Stem of "explanation": %s' %snowball.stem('explanation')
print 'Lancaster Stem of "explanation": %s' %lancaster.stem('explanation')
Explanation: You may notice from looking at this sample, however, that a potentially meaningful word has been removed: 'not'. This stopword corpus includes the words 'no', 'nor', and 'not'and so by removing these words we have removed negation. 
<h3>3. Stemming and Lemmatization</h3>
<b> I think we might want to beef up the explanation here a little bit more. Also, do we want to go into POS tagging? </b>
<p>The overarching goal of stemming and lemmatization is to reduce differential forms of a word to a common base form. This step will allow you to count occurrences of words in the vectorization step. In deciding how to reduce the differential forms of words, you will want to consider how much information you will need to retain for your application. For instance, in many cases markers of tense and plurality are not informative, and so removing these markers will allow you to reduce the number of features.</p>
<p> <b>Stemming</b> is the process of representing the word as its root word while removing inflection. For example, the stem of the word 'explained' is 'explain'. By passing this word through the stemmer you would remove the tense inflection. There are multiple approaches to stemming: Porter stemming, Porter2 (snowball) stemming, and Lancaster stemming. You can read more in depth about these approaches.</p>
End of explanation
from nltk.stem.wordnet import WordNetLemmatizer
lemmatizer = WordNetLemmatizer()
print lemmatizer.lemmatize('explanation')
Explanation: While <b><em>stemming</em></b> is a heuristic process that selectively removes the end of words, <b><em>lemmatization</em></b> is a more sophisticated process that takes into account variables such as part-of-speech, meaning, and context within a document or neighboring sentences.</p>
End of explanation
stemmed_tokens = []
lemmatized_tokens = []
for token in cleaned_tokens:
    stemmed_tokens.append(stemmer.stem(token))
    lemmatized_tokens.append(lemmatizer.lemmatize(token))
Explanation: <p>In this example, lemmatization retains a bit more information than stemming. Within stemming, the Lancaster method is more aggressive than Porter and Snowball. Remember that this step allows us to reduce words to a common base form so that we can reduce our feature space and perform counting of occurrences. It will depend on your data and your application as to how much information you need to retain. </p>
<p>See also: <a href=http://nlp.stanford.edu/IR-book/html/htmledition/stemming-and-lemmatization-1.html>Stemming and lemmatization</a></p>
Example: Stemming and Lemmatization
To illustrate the difference between stemming and lemmatization, we will apply both methods to our articles.
End of explanation
print stemmed_tokens[:50]
Explanation: Let's take a look at a sample of our stemmed tokens
End of explanation
print lemmatized_tokens[:50]
Explanation: In contrast, here are the same tokens in their lemmatized form
End of explanation
from sklearn.feature_extraction.text import CountVectorizer
vectorizer = CountVectorizer()
stemmed_article = ' '.join(wd for wd in stemmed_tokens)
article_vect = vectorizer.fit_transform([stemmed_article])
Explanation: <h3>4. Vectorization </h3>
<p> Often in natural language processing we want to represent our text as a quantitative set of features for subsequent analysis. One way to generate features from text is to count the occurrences words. This apporoach is often referred to as a bag of words approach.</p>
<p>In the example of our article, we could represent the article as a vector of counts for each token. If we did the same for all of the other articles, we would have a set of vectors with each vector representing an article. If we had only one article, then we could have split the article into sentences and then represented each sentence as a vector. </p>
<p>If we apply a count vectorizer to our article, we will have a vector with the length of the number of unique tokens. </p>
Example: Count Vectorization of Article
For this example we will use the stemmed tokens from our article. We will need to join the tokens together to represent one article.
Check out the documentation for CountVectorizer in scikit-learn. You will see that there are a number of parameters that you can specify - including the maximum number of features. Depending on your data, you may choose to restrict the number of features by removing words that appear with least frequency.
End of explanation
<END_TASK>