<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Adult Income Classification
In this example, we would try to classify if the income is high i.e. >50 k or low i.e. < 50 k.
The dataset can be downloaded from 
Step1:  The adult file itself contains two blank lines at the end of the file. By default, pandas
will interpret the penultimate new line to be an empty (but valid) row. To remove
this, we remove any line with invalid numbers (the use of inplace just makes sure
the same Dataframe is affected, rather than creating a new one)
Step2:  The results show each of the feature names that are stored inside an Index object
from pandas.
The Adult dataset contains several categorical features, with Work-Class being one
example. While we could argue that some values are of higher rank than others (for
instance, a person with a job is likely to have a better income than a person without),
it doesn't make sense for all values. For example, a person working for the state
government is not more or less likely to have a higher income than someone working
in the private sector.
We can view the unique values for this feature in the dataset using the
unique() function
Step3:  There are some missing values in the preceding dataset, but they won't affect our
computations in this example.
Selecting the best individual features
If we have a number of features, the problem of finding the best subset is a difficult
task. 
The scikit-learn package has a number of transformers for performing univariate
feature selection. They include SelectKBest, which returns the k best performing
features, and SelectPercentile, which returns the top r% of features. In both cases,
there are a number of methods of computing the quality of a feature.
There are many different methods to compute how effectively a single feature
correlates with a class value. A commonly used method is the chi-squared (χ2) test.
Other methods include mutual information and entropy.
We can observe single-feature tests in action using our Adult dataset. First, we
extract a dataset and class values from our pandas DataFrame. We get a selection of
the features
Step4:  We will also create a target class array by testing whether the Earnings-Raw value
is above $50,000 or not. If it is, the class will be True. Otherwise, it will be False.
Let's look at the code
Step5:  Next, we create our transformer using the chi2 function and a SelectKBest
transformer
Step6:  Running fit_transform will call fit and then transform with the same dataset.
The result will create a new dataset, choosing only the best three features.
Let's look at the code
Step7:  The resulting matrix now only contains three features. We can also get the scores
for each column, allowing us to find out which features were used. Let's look at
the code
Step8:  The highest values are for the first, third, and fourth columns Correlates to the Age,
Capital-Gain, and Capital-Loss features. Based on a univariate feature selection,
these are the best features to choose.
We could also implement other correlations, such as the Pearson's correlation
coefficient. This is implemented in SciPy, a library used for scientific computing
(scikit-learn uses it as a base).
Step9:  The preceding function almost fits the interface needed to be used in scikit-learn's
univariate transformers. The function needs to accept two arrays (x and y in our
example) as parameters and returns two arrays, the scores for each feature and the
corresponding p-values. The chi2 function we used earlier only uses the required
interface, which allowed us to just pass it directly to SelectKBest.
The pearsonr function in SciPy accepts two arrays; however, the X array it accepts is
only one dimension. We will write a wrapper function that allows us to use this for
multivariate arrays like the one we have. Let's look at the code
Step10:  The Pearson value could be between -1 and 1. A value of 1 implies a
perfect correlation between two variables, while a value of -1 implies a
perfect negative correlation, that is, high values in one variable give low
values in the other and vice versa. Such features are really useful to have,
but would be discarded. For this reason, we have stored the absolute
value in the scores array, rather than the original signed value.
Step11:  This returns a different set of features! The features chosen this way are the first,
second, and fifth columns
Step12:  It is worth remembering the goal of this data mining activity
Step13:  The result, array([ 0.997,  0.003,  0.   ,  0.   ,  0.   ]), shows
us that the first feature accounts for 99.7 percent of the variance in the dataset,
the second accounts for 0.3 percent, and so on. By the fourth feature, less than
one-tenth of a percent of the variance is contained in the feature. The other
features explain even less.
Step14:  
Step15:  
Step16:  Now all of these features are categorical classes, but most scikit-learn classifiers (in particular the Decision Trees we plan to use), expect real-valued attributes. We can easily convert sex  to a binary value (0=female,1=male). We will use the LabelEncoder class from scikit-learn
Step18:  Now, we have to convert other categorical features. Since we have three different classes, we cannot convert to binary values (and using 0/1/2 values would imply an order, something we do not want). We use pandas get_dummies method for it or we can also use the OneHotEncoder to get three different attributes
Step19:  Feature importances with forests of trees
This examples shows the use of forests of trees to evaluate the importance of features on an artificial classification task. The red bars are the feature importances of the forest, along with their inter-trees variability.
Step20:  feature_importances_
Step21:  Decision Tree accuracy and time elapsed caculation
Step22:  cross validation for DT
Step23:  Tuning our hyperparameters using GridSearch
Step24:  Random Forest accuracy and time elapsed caculation
Step25:  cross validation for RF
Step26:  Receiver Operating Characteristic (ROC) curve
Step27:  Tuning Models using GridSearch
Step28:  Naive Bayes accuracy and time elapsed caculation
Step29:  cross-validation for NB
Step30:  KNN accuracy and time elapsed caculation
Step31:  cross validation for KNN
<ASSISTANT_TASK:>
Python Code:
import os
from sklearn.tree import DecisionTreeClassifier, export_graphviz
import pandas as pd
import numpy as np
from sklearn.cross_validation import train_test_split
from sklearn import cross_validation, metrics
from sklearn.ensemble import RandomForestClassifier
from sklearn.naive_bayes import BernoulliNB
from sklearn.neighbors import KNeighborsClassifier
from sklearn.svm import SVC
from time import time
from sklearn.pipeline import Pipeline
from sklearn.metrics import roc_auc_score , classification_report
from sklearn.grid_search import GridSearchCV
from sklearn.pipeline import Pipeline
from sklearn.metrics import precision_score, recall_score, accuracy_score, classification_report
# read .csv from provided dataset
csv_filename="datasets/adult.data"
df = pd.read_csv(csv_filename, header=None,
                names=["Age", "Work-Class", "fnlwgt",
                "Education", "Education-Num",
                "Marital-Status", "Occupation",
                "Relationship", "Race", "Sex",
                "Capital-gain", "Capital-loss",
                "Hours-per-week", "Native-Country",
                "Earnings-Raw"])
df.replace([' <=50K',' >50K'],[0,1],inplace=True)
df.tail()
Explanation: Adult Income Classification
In this example, we would try to classify if the income is high i.e. >50 k or low i.e. < 50 k.
The dataset can be downloaded from : http://archive.ics.uci.edu/ml/datasets/Adult
Listing of attributes:
Class: >50K, <=50K. 
age: continuous. 
workclass: Private, Self-emp-not-inc, Self-emp-inc, Federal-gov, Local-gov, State-gov, Without-pay, Never-worked. 
fnlwgt: continuous. 
education: Bachelors, Some-college, 11th, HS-grad, Prof-school, Assoc-acdm, Assoc-voc, 9th, 7th-8th, 12th, Masters, 1st-4th, 10th, Doctorate, 5th-6th, Preschool. 
education-num: continuous. 
marital-status: Married-civ-spouse, Divorced, Never-married, Separated, Widowed, Married-spouse-absent, Married-AF-spouse. 
occupation: Tech-support, Craft-repair, Other-service, Sales, Exec-managerial, Prof-specialty, Handlers-cleaners, Machine-op-inspct, Adm-clerical, Farming-fishing, Transport-moving, Priv-house-serv, Protective-serv, Armed-Forces. 
relationship: Wife, Own-child, Husband, Not-in-family, Other-relative, Unmarried. 
race: White, Asian-Pac-Islander, Amer-Indian-Eskimo, Other, Black. 
sex: Female, Male. 
capital-gain: continuous. 
capital-loss: continuous. 
hours-per-week: continuous. 
native-country: United-States, Cambodia, England, Puerto-Rico, Canada, Germany, Outlying-US(Guam-USVI-etc), India, Japan, 
Greece, South, China, Cuba, Iran, Honduras, Philippines, Italy, Poland, Jamaica, Vietnam, Mexico, Portugal, Ireland, France, Dominican-Republic, Laos, Ecuador, Taiwan, Haiti, Columbia, Hungary, Guatemala, Nicaragua, Scotland, Thailand, Yugoslavia, El-Salvador, Trinadad&Tobago, Peru, Hong, Holand-Netherlands.
End of explanation
df.dropna(how='all', inplace=True)
feature_names = df.columns
Explanation: The adult file itself contains two blank lines at the end of the file. By default, pandas
will interpret the penultimate new line to be an empty (but valid) row. To remove
this, we remove any line with invalid numbers (the use of inplace just makes sure
the same Dataframe is affected, rather than creating a new one):
End of explanation
df["Work-Class"].unique()
Explanation: The results show each of the feature names that are stored inside an Index object
from pandas.
The Adult dataset contains several categorical features, with Work-Class being one
example. While we could argue that some values are of higher rank than others (for
instance, a person with a job is likely to have a better income than a person without),
it doesn't make sense for all values. For example, a person working for the state
government is not more or less likely to have a higher income than someone working
in the private sector.
We can view the unique values for this feature in the dataset using the
unique() function:
End of explanation
X = df[["Age", "Education-Num", "Capital-gain", "Capital-loss", "Hours-per-week"]].values
Explanation: There are some missing values in the preceding dataset, but they won't affect our
computations in this example.
Selecting the best individual features
If we have a number of features, the problem of finding the best subset is a difficult
task. 
The scikit-learn package has a number of transformers for performing univariate
feature selection. They include SelectKBest, which returns the k best performing
features, and SelectPercentile, which returns the top r% of features. In both cases,
there are a number of methods of computing the quality of a feature.
There are many different methods to compute how effectively a single feature
correlates with a class value. A commonly used method is the chi-squared (χ2) test.
Other methods include mutual information and entropy.
We can observe single-feature tests in action using our Adult dataset. First, we
extract a dataset and class values from our pandas DataFrame. We get a selection of
the features:
End of explanation
y = (df["Earnings-Raw"]).values
y
Explanation: We will also create a target class array by testing whether the Earnings-Raw value
is above $50,000 or not. If it is, the class will be True. Otherwise, it will be False.
Let's look at the code:
End of explanation
from sklearn.feature_selection import SelectKBest
from sklearn.feature_selection import chi2
transformer = SelectKBest(score_func=chi2, k=3)
Explanation: Next, we create our transformer using the chi2 function and a SelectKBest
transformer:
End of explanation
Xt_chi2 = transformer.fit_transform(X, y)
Explanation: Running fit_transform will call fit and then transform with the same dataset.
The result will create a new dataset, choosing only the best three features.
Let's look at the code:
End of explanation
print(transformer.scores_)
Explanation: The resulting matrix now only contains three features. We can also get the scores
for each column, allowing us to find out which features were used. Let's look at
the code:
End of explanation
from scipy.stats import pearsonr
Explanation: The highest values are for the first, third, and fourth columns Correlates to the Age,
Capital-Gain, and Capital-Loss features. Based on a univariate feature selection,
these are the best features to choose.
We could also implement other correlations, such as the Pearson's correlation
coefficient. This is implemented in SciPy, a library used for scientific computing
(scikit-learn uses it as a base).
End of explanation
def multivariate_pearsonr(X, y):
    scores, pvalues = [], []
    for column in range(X.shape[1]):
        cur_score, cur_p = pearsonr(X[:,column], y)
        scores.append(abs(cur_score))
        pvalues.append(cur_p)
    return (np.array(scores), np.array(pvalues))
Explanation: The preceding function almost fits the interface needed to be used in scikit-learn's
univariate transformers. The function needs to accept two arrays (x and y in our
example) as parameters and returns two arrays, the scores for each feature and the
corresponding p-values. The chi2 function we used earlier only uses the required
interface, which allowed us to just pass it directly to SelectKBest.
The pearsonr function in SciPy accepts two arrays; however, the X array it accepts is
only one dimension. We will write a wrapper function that allows us to use this for
multivariate arrays like the one we have. Let's look at the code:
End of explanation
transformer = SelectKBest(score_func=multivariate_pearsonr, k=3)
Xt_pearson = transformer.fit_transform(X, y)
print(transformer.scores_)
Explanation: The Pearson value could be between -1 and 1. A value of 1 implies a
perfect correlation between two variables, while a value of -1 implies a
perfect negative correlation, that is, high values in one variable give low
values in the other and vice versa. Such features are really useful to have,
but would be discarded. For this reason, we have stored the absolute
value in the scores array, rather than the original signed value.
End of explanation
from sklearn.tree import DecisionTreeClassifier
from sklearn.cross_validation import cross_val_score
clf = DecisionTreeClassifier(random_state=14)
scores_chi2 = cross_val_score(clf, Xt_chi2, y, scoring='accuracy')
scores_pearson = cross_val_score(clf, Xt_pearson, y, scoring='accuracy')
print("Chi2 performance: {0:.3f}".format(scores_chi2.mean()))
print("Pearson performance: {0:.3f}".format(scores_pearson.mean()))
Explanation: This returns a different set of features! The features chosen this way are the first,
second, and fifth columns: the Age, Education, and Hours-per-week worked. This
shows that there is not a definitive answer to what the best features are— it depends
on the metric.
We can see which feature set is better by running them through a classifier. Keep in
mind that the results only indicate which subset is better for a particular classifier
and/or feature combination—there is rarely a case in data mining where one method
is strictly better than another in all cases! Let's look at the code:
End of explanation
from sklearn.decomposition import PCA
pca = PCA(n_components=5)
Xd = pca.fit_transform(X)
np.set_printoptions(precision=3, suppress=True)
pca.explained_variance_ratio_
Explanation: It is worth remembering the goal of this data mining activity: predicting wealth.
Using a combination of good features and feature selection, we can achieve 83
percent accuracy using just three features of a person!
Principal Components Analysis
End of explanation
clf = DecisionTreeClassifier(random_state=14)
original_scores = cross_val_score(clf, X, y, scoring='accuracy')
print("The average score from the original dataset is {:.4f}".format(np.mean(original_scores)))
clf = DecisionTreeClassifier(random_state=14)
scores_reduced = cross_val_score(clf, Xd, y, scoring='accuracy')
print("The average score from the reduced dataset is {:.4f}".format(np.mean(scores_reduced)))
Explanation: The result, array([ 0.997,  0.003,  0.   ,  0.   ,  0.   ]), shows
us that the first feature accounts for 99.7 percent of the variance in the dataset,
the second accounts for 0.3 percent, and so on. By the fourth feature, less than
one-tenth of a percent of the variance is contained in the feature. The other
features explain even less.
End of explanation
df.replace(' ?',np.nan,inplace=True)
df.loc[26,:]
df.shape
#We can also specify to only drop rows that are complete missing all data
df.dropna(inplace=True)
df.shape
Explanation: 
End of explanation
df.head()
for col in df.columns:
    if df[col].dtype != 'int64':
        print ("For: {}, Total unique values are {} - \n {}".format(col ,
                                                        len(pd.Series(df[col].values.ravel()).unique()),
                                                        pd.Series(df[col].values.ravel()).unique()))
        print ('\n')
Explanation: 
End of explanation
from sklearn.preprocessing import LabelEncoder
enc = LabelEncoder()
label_encoder = enc.fit(df['Sex'])
print ("Categorical classes:", label_encoder.classes_)
integer_classes = label_encoder.transform(label_encoder.classes_)
print ("Integer classes:", integer_classes)
t = label_encoder.transform(df['Sex'])
df['Sex'] = t
print ('Feature names:',feature_names)
print ('Features:', df['Sex'][10:16])
Explanation: Now all of these features are categorical classes, but most scikit-learn classifiers (in particular the Decision Trees we plan to use), expect real-valued attributes. We can easily convert sex  to a binary value (0=female,1=male). We will use the LabelEncoder class from scikit-learn:
End of explanation
categorical_features = []
for col in df.columns:
    if df[col].dtype != 'int64':
        categorical_features.append(col)
categorical_features
df.shape[0]
onehot_df = pd.get_dummies(df)
onehot_df.head()
onehot_df.columns
from sklearn.preprocessing import OneHotEncoder
enc = LabelEncoder()
for f in categorical_features:
    label_encoder = enc.fit(df[f])
    print "Categorical classes:", label_encoder.classes_
    
    integer_classes = label_encoder.transform(label_encoder.classes_).reshape(
        len(pd.Series(df[f].values.ravel()).unique()), 1)
    print "Integer classes:", integer_classes
    
    enc = OneHotEncoder()
    one_hot_encoder = enc.fit(integer_classes)
    
    # First, convert clases to 0-(N-1) integers using label_encoder
    num_of_rows = df.shape[0]
    t = label_encoder.transform(df[f]).reshape(num_of_rows, 1)
    
    # Second, create a sparse matrix with three columns, each one indicating if the instance belongs to the class
    new_features = one_hot_encoder.transform(t)
    
    # Add the new features to df
    df = np.concatenate([df, new_features.toarray()], axis = 1)
    
    #Eliminate converted columns
    df.drop(f, axis=1, inplace=True)
    
    # Update feature names
    feature_names = ['age', 'sex', 'first_class', 'second_class', 'third_class']
    # Convert to numerical values
    df = df.astype(float)
    titanic_y = titanic_y.astype(float)
features_list = list(onehot_df.columns)
features_list.remove('Earnings-Raw')
X = onehot_df[features_list]
y= onehot_df['Earnings-Raw']
Explanation: Now, we have to convert other categorical features. Since we have three different classes, we cannot convert to binary values (and using 0/1/2 values would imply an order, something we do not want). We use pandas get_dummies method for it or we can also use the OneHotEncoder to get three different attributes:
End of explanation
# split dataset to 60% training and 40% testing
X_train, X_test, y_train, y_test = cross_validation.train_test_split(X, y, test_size=0.4, random_state=0)
print (X_train.shape, y_train.shape)
Explanation: Feature importances with forests of trees
This examples shows the use of forests of trees to evaluate the importance of features on an artificial classification task. The red bars are the feature importances of the forest, along with their inter-trees variability.
End of explanation
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
from sklearn.ensemble import ExtraTreesClassifier
# Build a classification task using 3 informative features
# Build a forest and compute the feature importances
forest = ExtraTreesClassifier(n_estimators=250,
                              random_state=0)
forest.fit(X, y)
importances = forest.feature_importances_
std = np.std([tree.feature_importances_ for tree in forest.estimators_],
             axis=0)
indices = np.argsort(importances)[::-1]
importances[indices[:5]]
features = features_list
for f in range(5):
    print("%d. feature %d - %s (%f)" % (f + 1, indices[f], features[indices[f]] ,importances[indices[f]]))
best_features = []
for i in indices[:5]:
    best_features.append(features[i])
# Plot the top 5 feature importances of the forest
plt.figure(num=None, figsize=(8, 6), dpi=80, facecolor='w', edgecolor='k')
plt.title("Feature importances")
plt.bar(range(5), importances[indices][:5], 
       color="r",  yerr=std[indices][:5], align="center")
plt.xticks(range(5), best_features)
plt.xlim([-1, 5])
plt.show()
Explanation: feature_importances_
End of explanation
t0=time()
print ("DecisionTree")
dt = DecisionTreeClassifier(min_samples_split=20,random_state=99)
# dt = DecisionTreeClassifier(min_samples_split=20,max_depth=5,random_state=99)
clf_dt=dt.fit(X_train,y_train)
print ("Acurracy: ", clf_dt.score(X_test,y_test))
t1=time()
print ("time elapsed: ", t1-t0)
Explanation: Decision Tree accuracy and time elapsed caculation
End of explanation
tt0=time()
print ("cross result========")
scores = cross_validation.cross_val_score(dt, X, y, cv=5)
print (scores)
print (scores.mean())
tt1=time()
print ("time elapsed: ", tt1-tt0)
Explanation: cross validation for DT
End of explanation
from sklearn.metrics import classification_report
pipeline = Pipeline([
    ('clf', DecisionTreeClassifier(criterion='entropy'))
])
parameters = {
    'clf__max_depth': (5, 25 , 50),
    'clf__min_samples_split': (2, 5, 10),
    'clf__min_samples_leaf': (1, 2, 3)
}
grid_search = GridSearchCV(pipeline, parameters, n_jobs=-1, verbose=1, scoring='f1')
grid_search.fit(X_train, y_train)
print ('Best score: %0.3f' % grid_search.best_score_)
print ('Best parameters set:')
best_parameters = grid_search.best_estimator_.get_params()
for param_name in sorted(parameters.keys()):
    print ('\t%s: %r' % (param_name, best_parameters[param_name]))
predictions = grid_search.predict(X_test)
print (classification_report(y_test, predictions))
Explanation: Tuning our hyperparameters using GridSearch
End of explanation
t2=time()
print ("RandomForest")
rf = RandomForestClassifier(n_estimators=100,n_jobs=-1)
clf_rf = rf.fit(X_train,y_train)
print ("Acurracy: ", clf_rf.score(X_test,y_test))
t3=time()
print ("time elapsed: ", t3-t2)
Explanation: Random Forest accuracy and time elapsed caculation
End of explanation
tt2=time()
print ("cross result========")
scores = cross_validation.cross_val_score(rf, X, y, cv=5)
print (scores)
print (scores.mean())
tt3=time()
print ("time elapsed: ", tt3-tt2)
Explanation: cross validation for RF
End of explanation
roc_auc_score(y_test,rf.predict(X_test))
%matplotlib inline
import matplotlib.pyplot as plt
from sklearn.metrics import roc_curve, auc
predictions = rf.predict_proba(X_test)
false_positive_rate, recall, thresholds = roc_curve(y_test, predictions[:, 1])
roc_auc = auc(false_positive_rate, recall)
plt.title('Receiver Operating Characteristic')
plt.plot(false_positive_rate, recall, 'b', label='AUC = %0.2f' % roc_auc)
plt.legend(loc='lower right')
plt.plot([0, 1], [0, 1], 'r--')
plt.xlim([0.0, 1.0])
plt.ylim([0.0, 1.0])
plt.ylabel('Recall')
plt.xlabel('Fall-out')
plt.show()
Explanation: Receiver Operating Characteristic (ROC) curve
End of explanation
pipeline2 = Pipeline([
('clf', RandomForestClassifier(criterion='entropy'))
])
parameters = {
    'clf__n_estimators': (5, 25, 50, 100),
    'clf__max_depth': (5, 25 , 50),
    'clf__min_samples_split': (2, 5, 10),
    'clf__min_samples_leaf': (1, 2, 3)
}
grid_search = GridSearchCV(pipeline2, parameters, n_jobs=-1, verbose=1, scoring='accuracy', cv=3)
grid_search.fit(X_train, y_train)
print ('Best score: %0.3f' % grid_search.best_score_)
print ('Best parameters set:')
best_parameters = grid_search.best_estimator_.get_params()
for param_name in sorted(parameters.keys()):
    print ('\t%s: %r' % (param_name, best_parameters[param_name]))
predictions = grid_search.predict(X_test)
print ('Accuracy:', accuracy_score(y_test, predictions))
print (classification_report(y_test, predictions))
    
Explanation: Tuning Models using GridSearch
End of explanation
t4=time()
print ("NaiveBayes")
nb = BernoulliNB()
clf_nb=nb.fit(X_train,y_train)
print ("Acurracy: ", clf_nb.score(X_test,y_test))
t5=time()
print ("time elapsed: ", t5-t4)
Explanation: Naive Bayes accuracy and time elapsed caculation
End of explanation
tt4=time()
print ("cross result========")
scores = cross_validation.cross_val_score(nb, X, y, cv=5)
print (scores)
print (scores.mean())
tt5=time()
print ("time elapsed: ", tt5-tt4)
Explanation: cross-validation for NB
End of explanation
t6=time()
print ("KNN")
# knn = KNeighborsClassifier(n_neighbors=3)
knn = KNeighborsClassifier()
clf_knn=knn.fit(X_train, y_train)
print ("Acurracy: ", clf_knn.score(X_test,y_test) )
t7=time()
print ("time elapsed: ", t7-t6)
Explanation: KNN accuracy and time elapsed caculation
End of explanation
tt6=time()
print ("cross result========")
scores = cross_validation.cross_val_score(knn, X, y, cv=5)
print (scores)
print (scores.mean())
tt7=time()
print ("time elapsed: ", tt7-tt6)
Explanation: cross validation for KNN
End of explanation
<END_TASK>