<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Load the Lending Club dataset
1. Load the dataset into a data frame named loans
Step1:  Exploring some features
2. print out the column names
Step2:  Exploring the target column
The target column (label column) of the dataset that we are interested in is called bad_loans. In this column 1 means a risky (bad) loan 0 means a safe loan.
In order to make this more intuitive and consistent with the lectures, we reassign the target to be
Step3:  4. Now, let us explore the distribution of the column safe_loans.
Step4:  Features for the classification algorithm
5. using a subset of features (categorical and numeric)
Step5:  One-hot encoding
Step6:  Build a decision tree classifier
Step7:  Visualizing a learned model (Optional)
Step8:  Making predictions
Let's consider two positive and two negative examples from the validation set and see what the model predicts. We will do the following
Step9:  12. Now, we will use our model to predict whether or not a loan is likely to default.
Step10:  Quiz Question 1
Step11:  Quiz Question 2
Step12:  Quiz Question 3
Step13:  17. Now, evaluate the accuracy of the small_model and decision_tree_model on the entire validation_data, not just the subsample considered above.
Step14:  Quiz Question 5
Step15:  19. Evaluate the accuracy of big_model on the training set and validation set.
Step16:  Quiz Question 6
<ASSISTANT_TASK:>
Python Code:
loans = pd.read_csv('lending-club-data.csv')
loans.head(2)
Explanation: Load the Lending Club dataset
1. Load the dataset into a data frame named loans
End of explanation
loans.columns
Explanation: Exploring some features
2. print out the column names
End of explanation
# safe_loans =  1 => safe
# safe_loans = -1 => risky
loans['safe_loans'] = loans['bad_loans'].apply(lambda x : +1 if x==0 else -1)
#loans = loans.remove_column('bad_loans')
loans = loans.drop('bad_loans', axis=1)
Explanation: Exploring the target column
The target column (label column) of the dataset that we are interested in is called bad_loans. In this column 1 means a risky (bad) loan 0 means a safe loan.
In order to make this more intuitive and consistent with the lectures, we reassign the target to be:
+1 as a safe loan
-1 as a risky (bad) loan
3. We put this in a new column called safe_loans.
End of explanation
loans['safe_loans'].value_counts()
print len(loans[loans['safe_loans']==+1]) *1.0/ len(loans['safe_loans'])
print len(loans[loans['safe_loans']==-1]) *1.0/ len(loans['safe_loans'])
Explanation: 4. Now, let us explore the distribution of the column safe_loans.
End of explanation
features = ['grade',                     # grade of the loan
            'sub_grade',                 # sub-grade of the loan
            'short_emp',                 # one year or less of employment
            'emp_length_num',            # number of years of employment
            'home_ownership',            # home_ownership status: own, mortgage or rent
            'dti',                       # debt to income ratio
            'purpose',                   # the purpose of the loan
            'term',                      # the term of the loan
            'last_delinq_none',          # has borrower had a delinquincy
            'last_major_derog_none',     # has borrower had 90 day or worse rating
            'revol_util',                # percent of available credit being used
            'total_rec_late_fee',        # total late fees received to day
           ]
target = 'safe_loans'                    # prediction target (y) (+1 means safe, -1 is risky)
# Extract the feature columns and target column
loans = loans[features + [target]]
Explanation: Features for the classification algorithm
5. using a subset of features (categorical and numeric)
End of explanation
categorical_variables = []
for feat_name, feat_type in zip(loans.columns, loans.dtypes):
    if feat_type == object:
        categorical_variables.append(feat_name)
        
for feature in categorical_variables:
    
    loans_one_hot_encoded = pd.get_dummies(loans[feature],prefix=feature)
    #print loans_one_hot_encoded
    
    loans = loans.drop(feature, axis=1)
    for col in loans_one_hot_encoded.columns:
        loans[col] = loans_one_hot_encoded[col]
    
print loans.head(2)        
print loans.columns
#pd.get_dummies(loans, prefix=['col1', 'col2'])
with open('module-5-assignment-1-train-idx.json') as train_data_file:    
    train_idx  = json.load(train_data_file)
with open('module-5-assignment-1-validation-idx.json') as validation_data_file:    
    validation_idx = json.load(validation_data_file)
print train_idx[:3]
print validation_idx[:3]
print len(train_idx)
print len(validation_idx)
train_data = loans.iloc[train_idx]
validation_data = loans.iloc[validation_idx]
print len(loans.dtypes )
Explanation: One-hot encoding
End of explanation
train_Y = train_data['safe_loans'].as_matrix()
train_X = train_data.drop('safe_loans', axis=1).as_matrix()
print train_Y.shape
print train_X.shape
from sklearn.tree import DecisionTreeClassifier
decision_tree_model = DecisionTreeClassifier(max_depth=6)
decision_tree_model = decision_tree_model.fit(train_X, train_Y)
small_model = DecisionTreeClassifier(max_depth=2)
small_model = small_model.fit(train_X, train_Y)
import sklearn
print('The scikit-learn version is {}.'.format(sklearn.__version__))
Explanation: Build a decision tree classifier
End of explanation
from sklearn import tree
import graphviz 
from os import system
dot_data = tree.export_graphviz(small_model, out_file='simple_tree.dot',
                               feature_names=train_data.columns,  
                         class_names=['+1','-1'],  
                         filled=True, rounded=True,  
                         special_characters=True) 
system("dot -Tpng simple_tree.dot -o simple_tree.png")
from IPython.display import Image
Image(filename='simple_tree.png') 
Explanation: Visualizing a learned model (Optional)
End of explanation
validation_safe_loans = validation_data[validation_data[target] == 1]
validation_risky_loans = validation_data[validation_data[target] == -1]
sample_validation_data_risky = validation_risky_loans[0:2]
sample_validation_data_safe = validation_safe_loans[0:2]
sample_validation_data = sample_validation_data_safe.append(sample_validation_data_risky)
sample_validation_data
Explanation: Making predictions
Let's consider two positive and two negative examples from the validation set and see what the model predicts. We will do the following:
Predict whether or not a loan is safe.
Predict the probability that a loan is safe.
11. First, let's grab 2 positive examples and 2 negative examples
End of explanation
sample_validation_data_Y = sample_validation_data['safe_loans'].as_matrix()
sample_validation_data_X = sample_validation_data.drop('safe_loans', axis=1).as_matrix()
print sample_validation_data_Y.shape
print sample_validation_data_X.shape
decision_tree_model.predict(sample_validation_data_X)
Explanation: 12. Now, we will use our model to predict whether or not a loan is likely to default.
End of explanation
decision_tree_model.predict_proba(sample_validation_data_X)
Explanation: Quiz Question 1:
What percentage of the predictions on sample_validation_data did decision_tree_model get correct?
Answer:
0.5
Explore probability predictions
13. For each row in the sample_validation_data, what is the probability (according decision_tree_model) of a loan being classified as safe? (Hint: if you are using scikit-learn, you can use the .predict_proba() method)
End of explanation
small_model.predict_proba(sample_validation_data_X)
Explanation: Quiz Question 2:
Which loan has the highest probability of being classified as a safe loan?
Answer:
4
Tricky predictions!
14. Now, we will explore something pretty interesting. For each row in the sample_validation_data, what is the probability (according to small_model) of a loan being classified as safe?
End of explanation
small_model.score(train_X, train_Y)
decision_tree_model.score(train_X, train_Y)
Explanation: Quiz Question 3:
Notice that the probability preditions are the exact same for the 2nd and 3rd loans. Why would this happen?
Quiz Question 4:
Based on the visualized tree, what prediction would you make for this data point (according to small_model)? (If you don't have Graphviz, you can answer this quiz question by executing the next part.)
15. Now, verify your prediction by examining the prediction made using small_model.
Evaluating accuracy of the decision tree model
16. Evaluate the accuracy of small_model and decision_tree_model on the training data. (Hint: if you are using scikit-learn, you can use the .score() method)
End of explanation
validation_Y = validation_data['safe_loans'].as_matrix()
validation_X = validation_data.drop('safe_loans', axis=1).as_matrix()
print validation_Y.shape
print validation_X.shape
small_model.score(validation_X, validation_Y)
decision_tree_model.score(validation_X, validation_Y)
Explanation: 17. Now, evaluate the accuracy of the small_model and decision_tree_model on the entire validation_data, not just the subsample considered above.
End of explanation
big_model = DecisionTreeClassifier(max_depth=10)
big_model = big_model.fit(train_X, train_Y)
Explanation: Quiz Question 5:
What is the accuracy of decision_tree_model on the validation set, rounded to the nearest .01?
Answer:
0.64
Evaluating accuracy of a complex decision tree model
18. Using sklearn.tree.DecisionTreeClassifier, train a decision tree with maximum depth = 10. Call this model big_model.
End of explanation
big_model.score(train_X, train_Y)
big_model.score(validation_X, validation_Y)
Explanation: 19. Evaluate the accuracy of big_model on the training set and validation set.
End of explanation
predictions = decision_tree_model.predict(validation_X)
print type(predictions)
print type(validation_Y)
false_positives = ((predictions==1) * (validation_Y==-1)).sum()
print false_positives
false_negatives = ((predictions==-1) * (validation_Y==1)).sum()
print false_negatives
correct_predictions = (predictions==validation_Y).sum()
print correct_predictions
print 10000 * false_negatives + 20000 * false_positives
Explanation: Quiz Question 6:
How does the performance of big_model on the validation set compare to decision_tree_model on the validation set? Is this a sign of overfitting?
Answer:
Yes
Quantifying the cost of mistakes
End of explanation
<END_TASK>