<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Udacity Machine Learning Feature Selection mini-project
Prep stuff
Step1:  Overfitting a decision tree (many features with few data points)
Step2:  Evaluating
Step3:  Finding particularly relevant features
Step4:  Identifying problematic word in question
Step5:  This looks rather like part of someone's email signature.
Loading modified data set where this has been pruned out
Step6:  Decision Tree trained on modified data
Step7:  New accuracy is still very high
Step8:  Still has a couple of possible problem words
Step9:  First one looks like a timestamp, second seems to be signature. Removing signature.
Pruning and reloading again
Step10:  Notable drop in accuracy
Step11:  Checking for overly-influential words
Step12:  These look like (probably) content
<ASSISTANT_TASK:>
Python Code:
import pickle
import numpy
from sklearn import cross_validation
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.tree import DecisionTreeClassifier
numpy.random.seed(42)
words_file = "../text_learning/your_word_data.pkl"
authors_file = "../text_learning/your_email_authors.pkl"
word_data = pickle.load( open(words_file, "r"))
authors = pickle.load( open(authors_file, "r") )
features_train, features_test, labels_train, labels_test = cross_validation.train_test_split(word_data, authors, test_size=0.1, random_state=42)
vectorizer = TfidfVectorizer(sublinear_tf=True, max_df=0.5,
                             stop_words='english')
features_train = vectorizer.fit_transform(features_train)
features_test  = vectorizer.transform(features_test).toarray()
print 'training data:',features_train.shape
print 'testing data:',features_test.shape
features_train = features_train[:150].toarray()
labels_train   = labels_train[:150]
Explanation: Udacity Machine Learning Feature Selection mini-project
Prep stuff:
End of explanation
clfOverfit = DecisionTreeClassifier()
clfOverfit.fit(features_train,labels_train)
Explanation: Overfitting a decision tree (many features with few data points)
End of explanation
clfOverfit.score(features_test,labels_test)
Explanation: Evaluating:
End of explanation
for i in range(clfOverfit.feature_importances_.shape[0]):
    if clfOverfit.feature_importances_[i] > 0.1:
        print 'feature',i,':',clfOverfit.feature_importances_[i]
Explanation: Finding particularly relevant features
End of explanation
vectorizer.get_feature_names()[33614]
Explanation: Identifying problematic word in question:
End of explanation
words_file = "../text_learning/your_word_data2.pkl"
authors_file = "../text_learning/your_email_authors2.pkl"
word_data = pickle.load( open(words_file, "r"))
authors = pickle.load( open(authors_file, "r") )
features_train, features_test, labels_train, labels_test = cross_validation.train_test_split(word_data, authors, test_size=0.1, random_state=42)
vectorizer = TfidfVectorizer(sublinear_tf=True, max_df=0.5,
                             stop_words='english')
features_train = vectorizer.fit_transform(features_train)
features_test  = vectorizer.transform(features_test).toarray()
features_train = features_train[:150].toarray()
labels_train   = labels_train[:150]
Explanation: This looks rather like part of someone's email signature.
Loading modified data set where this has been pruned out:
End of explanation
clfOverfit = DecisionTreeClassifier()
clfOverfit.fit(features_train,labels_train)
Explanation: Decision Tree trained on modified data:
End of explanation
clfOverfit.score(features_test,labels_test)
Explanation: New accuracy is still very high
End of explanation
for i in range(clfOverfit.feature_importances_.shape[0]):
    if clfOverfit.feature_importances_[i] > 0.1:
        print 'feature',i,':',clfOverfit.feature_importances_[i]
print vectorizer.get_feature_names()[8674]
print vectorizer.get_feature_names()[14343]
Explanation: Still has a couple of possible problem words:
End of explanation
words_file = "../text_learning/your_word_data3.pkl"
authors_file = "../text_learning/your_email_authors3.pkl"
word_data = pickle.load( open(words_file, "r"))
authors = pickle.load( open(authors_file, "r") )
features_train, features_test, labels_train, labels_test = cross_validation.train_test_split(word_data, authors, test_size=0.1, random_state=42)
vectorizer = TfidfVectorizer(sublinear_tf=True, max_df=0.5,
                             stop_words='english')
features_train = vectorizer.fit_transform(features_train)
features_test  = vectorizer.transform(features_test).toarray()
features_train = features_train[:150].toarray()
labels_train   = labels_train[:150]
clfOverfit = DecisionTreeClassifier()
clfOverfit.fit(features_train,labels_train)
Explanation: First one looks like a timestamp, second seems to be signature. Removing signature.
Pruning and reloading again
End of explanation
clfOverfit.score(features_test,labels_test)
Explanation: Notable drop in accuracy:
End of explanation
for i in range(clfOverfit.feature_importances_.shape[0]):
    if clfOverfit.feature_importances_[i] > 0.1:
        print 'feature',i,':',clfOverfit.feature_importances_[i]
Explanation: Checking for overly-influential words
End of explanation
print vectorizer.get_feature_names()[11975]
print vectorizer.get_feature_names()[18849]
print vectorizer.get_feature_names()[21323]
Explanation: These look like (probably) content
End of explanation
<END_TASK>