<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Intuition - Decision Trees
Decision Trees are a non-parametric supervised learning method used for classification and regression. The goal is to create a model that predicts the value of a target variable by learning simple decision rules inferred from the data features.
Paper Exercise
Let us start with a simple exercise in classifying credit risk. 
We have the following features in our dataset. 
- Risk - ordinal (label)
- Income - continuous
- Credit History - ordinal
We want to find out the rules that would help us classify the three risk type - This is a paper and pen exercise first!!
Step1:  Plotting the Data
Step2:  Preparing Data
We have one ordinal variable (Risk) and one nominal variable (Credit History)
Lets use LabelEncoder for the nominal variable
Step3:  Lets use a dictionary for encoding nominal variable
Step4:  Decision Tree Classifier
Step5:  Visualise the Tree
Step6:  Now we need to first define an Impurity measure. The three popular impurity measures are
<ASSISTANT_TASK:>
Python Code:
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
plt.style.use('fivethirtyeight')
plt.rcParams['figure.figsize'] = (9,6)
df = pd.read_csv("data/creditRisk.csv")
df.head()
Explanation: Intuition - Decision Trees
Decision Trees are a non-parametric supervised learning method used for classification and regression. The goal is to create a model that predicts the value of a target variable by learning simple decision rules inferred from the data features.
Paper Exercise
Let us start with a simple exercise in classifying credit risk. 
We have the following features in our dataset. 
- Risk - ordinal (label)
- Income - continuous
- Credit History - ordinal
We want to find out the rules that would help us classify the three risk type - This is a paper and pen exercise first!!
End of explanation
import seaborn as sns
sns.stripplot(data = df, x = "Income", y = "Credit History", hue = "Risk", size = 10)
Explanation: Plotting the Data
End of explanation
from sklearn import preprocessing
le = preprocessing.LabelEncoder()
df['Credit History'].unique()
df['Credit History'].unique()
le.fit(df['Credit History'].unique())
df['Credit History'].tail()
# Converting the categorical data using label encoder
df['Credit History'] = le.transform(df['Credit History'])
df['Credit History'].tail()
le.classes_
Explanation: Preparing Data
We have one ordinal variable (Risk) and one nominal variable (Credit History)
Lets use LabelEncoder for the nominal variable
End of explanation
df.Risk.unique()
Risk_mapping = {
       'High': 2,
       'Moderate': 1,
       'Low': 0}
df.Risk.tail()
df['Risk'] = df['Risk'].map(Risk_mapping)
df.Risk.tail()
df.head()
Explanation: Lets use a dictionary for encoding nominal variable
End of explanation
data = df.iloc[:,0:2]
target = df.iloc[:,2:3]
from sklearn import tree
clf = tree.DecisionTreeClassifier()
clf
clf = clf.fit(data, target)
Explanation: Decision Tree Classifier
End of explanation
import pydotplus 
from IPython.display import Image
dot_data = tree.export_graphviz(clf, out_file='tree.dot', feature_names=data.columns,
                                class_names=['Low', 'Moderate', 'High'], filled=True, 
                                rounded=True, special_characters=True)
graph = pydotplus.graph_from_dot_file('tree.dot')  
Image(graph.create_png()) 
import modelvis
modelvis.render_tree(clf, feature_names=data.columns,class_names=['Low', 'Moderate', 'High'] )
modelvis.print_tree_as_code(clf)
Explanation: Visualise the Tree
End of explanation
def plot_classifier_2d(clf, data, target):
    x_min, x_max = data.iloc[:,0].min(), data.iloc[:,0].max()
    y_min, y_max = data.iloc[:,1].min(), data.iloc[:,1].max()
    xx, yy = np.meshgrid(
        np.arange(x_min, x_max, (x_max - x_min)/100), 
        np.arange(y_min, y_max, (y_max - y_min)/100))
    Z = clf.predict(np.c_[xx.ravel(), yy.ravel()])
    Z = Z.reshape(xx.shape)
    cs = plt.contourf(xx, yy, Z, cmap="viridis", alpha = 0.5)
    plt.colorbar(cs)
    plt.scatter(x = data.iloc[:,0], y = data.iloc[:,1], c = target, s = 100, cmap="magma")
modelvis.plot_decision_boundaries(clf, data, target, feature_names=data.columns, show_input=True,
                                  class_names=['Low', 'Moderate', 'High'])
plot_classifier_2d(clf, data,target)
Explanation: Now we need to first define an Impurity measure. The three popular impurity measures are:
    - Gini Impurity
    - Entropy
    - Classification Error
Gini Impurity and Entropy lead to similiar results when growing the tree, while Classification error is not as useful for growing the tree (but for pruning the tree) - See example here http://sebastianraschka.com/faq/docs/decision-tree-binary.html
Lets understand Gini Impurity a little better. Gini impurity is a measure of how often a randomly chosen element from the set would be incorrectly labeled if it was randomly labeled according to the distribution of labels in the subset,  Gini impurity can be computed by summing the probability $t_{i} $ of an item with label $i$ being chosen times the probability $ 1-t_{i}$ of a mistake in categorizing that item.
$$ I_{G}(f)=\sum {i=1}^{J}t{i}(1-t_{i})=\sum {i=1}^{J}(t{i}-{t_{i}}^{2})=\sum {i=1}^{J}t{i}-\sum {i=1}^{J}{t{i}}^{2}=1-\sum {i=1}^{J}{t{i}}^{2} $$
Lets calculate the Gini for the overall data set:
Low - 4, Moderate - 6, High - 8 and total observations are 18 
$$ I_G(t) = 1 - \left(\frac{6}{18}\right)^2 - \left(\frac{4}{18}\right)^2 - \left(\frac{8}{18}\right)^2 = 1 - \frac{116}{256} = 0.642 $$
scikit-learn uses an optimized CART algorithm, which will use a greedy approach.  A greedy approach is used to divide the space called recursive binary splitting. This is a numerical procedure where all the values are lined up and different split points are tried and tested using a objective cost function. The split with the best cost (lowest cost because we minimize cost) is selected.
Another way to think of this is that a learned binary tree is actually a partitioning of the input space. You can think of each input variable as a dimension on an p-dimensional space. The decision tree split this up into rectangles (when p=2 input variables) or some kind of hyper-rectangles with more inputs. 
We can draw these partitions for our dataset
End of explanation
<END_TASK>