<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Basic usage of Sklearn
Step1:  Text processing with Scikit learn
We can use CountVectorizer to extract a bag of words representation from a collection of documents, using the SciKit-Learn method fit_transform. We will use a list of strings as documents.
Step2:  Array vector for the first document
Step3:  Number of times word "hard" occurs
Step4:  Using the 20 Newsgroups dataset
We are going to fetch just some categories so that it doesn't take that long to download the docs.
Step5:  Creating a CountVectorizer object
Step6:  We can now see how frequently the word algorithm occurs in the subset of the 20Newgroups collection we are considering.
Step7:  How many terms were extracted? use get_feature_names()
Step8:  CountVectorizer can do more preprocessing. This can be stopword removal.
Step9:  More preprocessing
For stemming and more advanced preprocessing, supplement SciKit Learn with another Python library, NLTK. Up next.
More advanced preprocessing with NLTK
NLTK is described in detail in a book by Bird, Klein and Loper available online
Step10:  Create an English stemmer
http
Step11:  NLTK for text analytics
NERs
Sentiment analysis
Extracting information from social media.
Step12:  Integrating NLTK with SciKit's vectorizer
NLTK Stemmer
The stemmer can be used to stem documents before feeding into SciKit's vectorizer, thus obtaining a more compact index.
One way to do this is to define a new class StemmedCountVectorizer extending CountVectorizer by redifining the method build_analyzer() that handles preprocessing and tokenization.
http
Step13:  If we modify build_analyzer() to apply the NLTK stemmer to the output of default build_analyzer(), we get a version that does stemming as well
Step14:  So now we can create an instance of this class
Step15:  Use this vectorizer to extract features
Compare this result to around 35,000 features we obtained using the unstemmed version.
Step16:  Notes
You should always experiment and see if it is good to use stemming with your problem set. It might not be the best thing to do.
SOLR works for processing larger datasets, since Python and SciKit-Learn become less effective, and more industrial strength software is required. One example of such software is Apache SOLR, an open source indexing package available from
<ASSISTANT_TASK:>
Python Code:
import sklearn
import numpy as np
import matplotlib.pyplot as plt
data = np.array([[1,2], [2,3], [3,4], [4,5], [5,6]])
x = data[:,0]
y = data[:,1]
data, x, y
Explanation: Basic usage of Sklearn
End of explanation
from sklearn.feature_extraction.text import CountVectorizer
vectorizer = CountVectorizer(min_df = 1)
content = ["How to format my hard disk", " Hard disk format problems "]
# fit_transform returns array of two rows, one per 'document'.
# each row has 7 elements, each element being the number of items
# a given feature occurred in that document.
X = vectorizer.fit_transform(content)
vectorizer.get_feature_names(), X.toarray()
Explanation: Text processing with Scikit learn
We can use CountVectorizer to extract a bag of words representation from a collection of documents, using the SciKit-Learn method fit_transform. We will use a list of strings as documents.
End of explanation
X.toarray()[0]
Explanation: Array vector for the first document
End of explanation
X.toarray()[1][vectorizer.get_feature_names().index('hard')]
Explanation: Number of times word "hard" occurs
End of explanation
from sklearn.datasets import fetch_20newsgroups
categories = ['alt.atheism', 'soc.religion.christian',
              'comp.graphics', 'sci.med']
twenty_train = fetch_20newsgroups(subset='train',
                                 categories=categories, shuffle=True,
                                 random_state=42)
Explanation: Using the 20 Newsgroups dataset
We are going to fetch just some categories so that it doesn't take that long to download the docs.
End of explanation
from sklearn.feature_extraction.text import CountVectorizer
vectorizer = CountVectorizer()
train_counts = vectorizer.fit_transform(twenty_train.data)
Explanation: Creating a CountVectorizer object
End of explanation
vectorizer.vocabulary_.get(u'algorithm')
Explanation: We can now see how frequently the word algorithm occurs in the subset of the 20Newgroups collection we are considering.
End of explanation
len(vectorizer.get_feature_names())
Explanation: How many terms were extracted? use get_feature_names()
End of explanation
vectorizer = CountVectorizer(stop_words='english')
sorted(vectorizer.get_stop_words())[0:20]
Explanation: CountVectorizer can do more preprocessing. This can be stopword removal.
End of explanation
import nltk
Explanation: More preprocessing
For stemming and more advanced preprocessing, supplement SciKit Learn with another Python library, NLTK. Up next.
More advanced preprocessing with NLTK
NLTK is described in detail in a book by Bird, Klein and Loper available online:
http://www.nltk.org/book_1ed/ for version 2.7 of python
About NLTK
It is not the best
It is very easy to use
You should read the book linked above to get familiar with the package and with text preprocessing.
End of explanation
s = nltk.stem.SnowballStemmer('english')
s.stem("cats"), s.stem("ran"), s.stem("jumped")
Explanation: Create an English stemmer
http://www.nltk.org/howto/stem.html for general intro.
http://www.nltk.org/api/nltk.stem.html for more details (including languages covered).
End of explanation
from nltk.tokenize import word_tokenize
text = word_tokenize("And now for something completely different")
nltk.pos_tag(text)
Explanation: NLTK for text analytics
NERs
Sentiment analysis
Extracting information from social media.
End of explanation
from sklearn.feature_extraction.text import CountVectorizer
vectorizer = CountVectorizer(stop_words="english")
analyze = vectorizer.build_analyzer()
analyze("John bought carrots and potatoes")
Explanation: Integrating NLTK with SciKit's vectorizer
NLTK Stemmer
The stemmer can be used to stem documents before feeding into SciKit's vectorizer, thus obtaining a more compact index.
One way to do this is to define a new class StemmedCountVectorizer extending CountVectorizer by redifining the method build_analyzer() that handles preprocessing and tokenization.
http://scikitlearn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html
build_analyzer() takes a string as input and outputs a list of tokens.
End of explanation
import nltk.stem
english_stemmer = nltk.stem.SnowballStemmer('english')
class StemmedCountVectorizer(CountVectorizer):
    def build_analyzer(self):
        analyzer=super(StemmedCountVectorizer, self).build_analyzer()
        return lambda doc:(english_stemmer.stem(w) for w in analyzer(doc))
Explanation: If we modify build_analyzer() to apply the NLTK stemmer to the output of default build_analyzer(), we get a version that does stemming as well:
End of explanation
stem_vectorizer = StemmedCountVectorizer(min_df=1,
                                        stop_words='english')
stem_analyze = stem_vectorizer.build_analyzer()
Y = stem_analyze("John bought carrots and potatoes")
[tok for tok in Y]
Explanation: So now we can create an instance of this class:
End of explanation
from sklearn.datasets import fetch_20newsgroups
categories = ['alt.atheism', 'soc.religion.christian',
             'comp.graphics', 'sci.med']
twenty_train = fetch_20newsgroups(subset='train',
                                 categories=categories,
                                 shuffle=True, random_state=42)
train_counts = stem_vectorizer.fit_transform(twenty_train.data)
len(stem_vectorizer.get_feature_names())
Explanation: Use this vectorizer to extract features
Compare this result to around 35,000 features we obtained using the unstemmed version.
End of explanation
!ipython nbconvert --to script Lab1\ Text\ processing\ with\ python.ipynb
Explanation: Notes
You should always experiment and see if it is good to use stemming with your problem set. It might not be the best thing to do.
SOLR works for processing larger datasets, since Python and SciKit-Learn become less effective, and more industrial strength software is required. One example of such software is Apache SOLR, an open source indexing package available from: 
http://lucene.apache.org/solr/
It produces Lucene-style indices that can be used by text analytics packages such as Mahout.
Elastic http://www.elastic.co/
End of explanation
<END_TASK>