<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Lesson 10 - Text Learning
When learning from text the biggest problem is that different text have different length. A smaller email would require lesser features while longer email would require more features.
Bag of words
Make a dictonary of counts of all the words that we care about.
word order does not matter
long phrases give different vectors
complex phrases cannot be handled like "chicago bulls"
Step1:  Not all words are equal
like the, hi etc.
stopwords
occur very frequently, low information and should be removed
Step2:  Not all unique words different
unresponsive
response
responsivity
responsiveness
respond
All of them can be passed through to get a root/stem - respon
We don't need all of them as their meaning is only slightly different and we don't get information.
<ASSISTANT_TASK:>
Python Code:
from sklearn.feature_extraction.text import CountVectorizer
vectorizer = CountVectorizer()
string1 = "hi aseem the car will be late regards company"
string2 = "hi company why will it be late I paid in advance regards aseem"
string3 = "hi aseem we don't know why will it be late regards company of company"
email_list = [string1, string2, string3]
vectorizer.fit(email_list)
bag_of_words = vectorizer.transform(email_list)
print vectorizer.vocabulary_
print bag_of_words
Explanation: Lesson 10 - Text Learning
When learning from text the biggest problem is that different text have different length. A smaller email would require lesser features while longer email would require more features.
Bag of words
Make a dictonary of counts of all the words that we care about.
word order does not matter
long phrases give different vectors
complex phrases cannot be handled like "chicago bulls"
End of explanation
import nltk
nltk.download()
from nltk.corpus import stopwords
sw = stopwords.words("english")
len(sw)
Explanation: Not all words are equal
like the, hi etc.
stopwords
occur very frequently, low information and should be removed
End of explanation
from nltk.stem.snowball import SnowballStemmer
stemmer = SnowballStemmer("english")
print stemmer.stem("responsiveness")
print stemmer.stem("responsivity")
print stemmer.stem("unresponsive")
Explanation: Not all unique words different
unresponsive
response
responsivity
responsiveness
respond
All of them can be passed through to get a root/stem - respon
We don't need all of them as their meaning is only slightly different and we don't get information.
End of explanation
<END_TASK>