<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Feature Extraction and Preprocessing
Step1:  DictVectorizer
Step2:  CountVectorizer
Step3:  Stop Word Filtering
Step4:  Stemming and Lemmatization   
Lemmatization is the process of determining the lemma, or the morphological root, of an inflected word based on its context. Lemmas are the base forms of words that are used to key the word in a dictionary.
Stemming has a similar goal to lemmatization, but it does not attempt to produce the morphological roots of words. Instead, stemming removes all patterns of characters that appear to be affixes, resulting in a token that is not necessarily a valid word.
Lemmatization frequently requires a lexical resource, like WordNet, and the word's part of speech. Stemming 
algorithms frequently use rules instead of lexical resources to produce stems and can 
operate on any token, even without its context.
Step5:  As we can see both sentences are having same meaning but their feature vectors have no elements in common. Let's use the lexical analysis on the data
Step6:  The Porter stemmer cannot consider the inflected form's part of speech and returns gather for both documents
Step7:  Extending bag-of-words with TF-IDF weights
It is intuitive that the frequency with which a word appears in a document could indicate the extent to which a document pertains to that word. A long document that contains one occurrence of a word may discuss an entirely different topic than a document that contains many occurrences of the same word. In this section, we will create feature vectors that encode the frequencies of words, and discuss strategies to mitigate two problems caused by encoding term frequencies.
      Instead of using a binary value for each element in the feature vector, we will now use an integer that represents the number of times that the words appeared in the document.
Step8:  Data Standardization
<ASSISTANT_TASK:>
Python Code:
from sklearn.feature_extraction import DictVectorizer
from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer, HashingVectorizer
from sklearn.metrics.pairwise import euclidean_distances
from sklearn import preprocessing
from nltk.stem.wordnet import WordNetLemmatizer
from nltk.stem import PorterStemmer
from nltk import word_tokenize
from nltk import pos_tag
import numpy as np
Explanation: Feature Extraction and Preprocessing
End of explanation
onehot_encoder = DictVectorizer()
instances = [
    {'city': 'New York'},
    {'city': 'San Francisco'},
    {'city': 'Chapel Hill'} ]
print (onehot_encoder.fit_transform(instances).toarray())
Explanation: DictVectorizer
End of explanation
corpus = [
    'UNC played Duke in basketball',
    'Duke lost the basketball game'
]
vectorizer = CountVectorizer()
print (vectorizer.fit_transform(corpus).todense())
print (vectorizer.vocabulary_)
# adding one more sentence in corpus
corpus = [
    'UNC played Duke in basketball',
    'Duke lost the basketball game',
    'This is Atul Singh'
]
vectorizer = CountVectorizer()
print (vectorizer.fit_transform(corpus).todense())
print (vectorizer.vocabulary_)
# checking the euclidean distance 
# converting sentence into CountVectorizer
counts = vectorizer.fit_transform(corpus).todense()
print("1 & 2", euclidean_distances(counts[0], counts[1]))
print("2 & 3", euclidean_distances(counts[1], counts[2]))
print("1 & 3", euclidean_distances(counts[0], counts[2]))
Explanation: CountVectorizer
End of explanation
vectorizer = CountVectorizer(stop_words='english')  # added one option which remove the grammer words from corpus
print (vectorizer.fit_transform(corpus).todense())
print (vectorizer.vocabulary_)
print("1 & 2", euclidean_distances(counts[0], counts[1]))
print("2 & 3", euclidean_distances(counts[1], counts[2]))
print("1 & 3", euclidean_distances(counts[0], counts[2]))
Explanation: Stop Word Filtering
End of explanation
corpus = [
    'He ate the sandwiches',
    'Every sandwich was eaten by him'
]
vectorizer = CountVectorizer(stop_words='english')  # added one option which remove the grammer words from corpus
print (vectorizer.fit_transform(corpus).todense())
print (vectorizer.vocabulary_)
Explanation: Stemming and Lemmatization   
Lemmatization is the process of determining the lemma, or the morphological root, of an inflected word based on its context. Lemmas are the base forms of words that are used to key the word in a dictionary.
Stemming has a similar goal to lemmatization, but it does not attempt to produce the morphological roots of words. Instead, stemming removes all patterns of characters that appear to be affixes, resulting in a token that is not necessarily a valid word.
Lemmatization frequently requires a lexical resource, like WordNet, and the word's part of speech. Stemming 
algorithms frequently use rules instead of lexical resources to produce stems and can 
operate on any token, even without its context.
End of explanation
lemmatizer = WordNetLemmatizer()
print (lemmatizer.lemmatize('gathering', 'v'))
print (lemmatizer.lemmatize('gathering', 'n'))
Explanation: As we can see both sentences are having same meaning but their feature vectors have no elements in common. Let's use the lexical analysis on the data
End of explanation
stemmer = PorterStemmer()
print (stemmer.stem('gathering'))
wordnet_tags = ['n', 'v']
corpus = [
'He ate the sandwiches',
'Every sandwich was eaten by him'
]
stemmer = PorterStemmer()
print ('Stemmed:', [[stemmer.stem(token) for token in word_tokenize(document)] for document in corpus])
def lemmatize(token, tag):
	if tag[0].lower() in ['n', 'v']:
		return lemmatizer.lemmatize(token, tag[0].lower())
	return token
lemmatizer = WordNetLemmatizer()
tagged_corpus = [pos_tag(word_tokenize(document)) for document in corpus]
print ('Lemmatized:', [[lemmatize(token, tag) for token, tag in document] for document in tagged_corpus])
Explanation: The Porter stemmer cannot consider the inflected form's part of speech and returns gather for both documents:
End of explanation
corpus = ['The dog ate a sandwich, the wizard transfigured a sandwich, and I ate a sandwich']
vectorizer = CountVectorizer(stop_words='english')
print (vectorizer.fit_transform(corpus).todense())
print(vectorizer.vocabulary_)
corpus = ['The dog ate a sandwich and I ate a sandwich',
    'The wizard transfigured a sandwich']
vectorizer = TfidfVectorizer(stop_words='english')
print (vectorizer.fit_transform(corpus).todense())
print(vectorizer.vocabulary_)
corpus = ['The dog ate a sandwich and I ate a sandwich',
    'The wizard transfigured a sandwich']
vectorizer = HashingVectorizer(n_features=6)
print (vectorizer.fit_transform(corpus).todense())
Explanation: Extending bag-of-words with TF-IDF weights
It is intuitive that the frequency with which a word appears in a document could indicate the extent to which a document pertains to that word. A long document that contains one occurrence of a word may discuss an entirely different topic than a document that contains many occurrences of the same word. In this section, we will create feature vectors that encode the frequencies of words, and discuss strategies to mitigate two problems caused by encoding term frequencies.
      Instead of using a binary value for each element in the feature vector, we will now use an integer that represents the number of times that the words appeared in the document.
End of explanation
X = [[1,2,3],
     [4,5,1],
     [3,6,2]
    ]
print(preprocessing.scale(X))
x1 = preprocessing.StandardScaler()
print(x1)
print(x1.fit_transform(X))
Explanation: Data Standardization
End of explanation
<END_TASK>