<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Key Requirements for the iRF scikit-learn implementation
The following is a documentation of the main requirements for the iRF implementation
Typical Setup
Import the required dependencies
In particular irf_utils and irf_jupyter_utils
Step1:  Step 1
Step2:  Check out the data
Step3:  Step 2
Step4:  STEP 3
Step5:  Perform Manual CHECKS on the irf_utils
These should be converted to unit tests and checked with nosetests -v test_irf_utils.py
Step 4
Step6:  Plot Ranked Feature Importances
Step7:  Decision Tree 0 (First) - Get output
Check the output against the decision tree graph
Step8:  Compare to our dict of extracted data from the tree
Step9:  Check output against the diagram
Step10:  Run the iRF function
We will run the iRF with the following parameters
Data
Step11:  Examine the stability scores
Step12:  That's interesting - feature 22, 27, 20, 23 keep popping up!
We should probably look at the feature importances to understand if there is a useful correlation
Examine feature importances
In particular, let us see how they change over the K iterations of random forest
Step13:  Some Observations
Note that after 5 iterations, the most important features were found to be 22, 27, 7, and 23
Now also recall that the most stable interactions were found to be '22_27', '7_22', '7_22_27', '23_27', '7_27', '22_23_27'
Given the overlap between these two plots, the results are not unreasonable here. 
Explore iRF Data Further
We can look at the decision paths of the Kth RF
Let's look at the final iteration RF - the key validation metrics
Step14:  We can get this data quite easily in a convenient format
Step15:  This checks nicely against the plotted diagram above.
In fact - we can go further and plot some interesting data from the Decision Trees
- This can help us understand variable interactions better
Step16:  We can also look at the frequency that a feature appears along a decision path
Step17:  The most common features that appeared were 27,22,23, and 7. This matches well with the feature importance plot above. 
Run some Sanity Checks
Run iRF for just 1 iteration - should be the uniform sampling version
This is just a sanity check
Step18:  Compare to the original single fitted random forest
<ASSISTANT_TASK:>
Python Code:
%matplotlib inline
import matplotlib.pyplot as plt
from sklearn.datasets import load_breast_cancer
import numpy as np
from functools import reduce
# Needed for the scikit-learn wrapper function
from sklearn.utils import resample
from sklearn.ensemble import RandomForestClassifier
from math import ceil
# Import our custom utilities
from imp import reload
from utils import irf_jupyter_utils
from utils import irf_utils
reload(irf_jupyter_utils)
reload(irf_utils)
Explanation: Key Requirements for the iRF scikit-learn implementation
The following is a documentation of the main requirements for the iRF implementation
Typical Setup
Import the required dependencies
In particular irf_utils and irf_jupyter_utils
End of explanation
load_breast_cancer = load_breast_cancer()
X_train, X_test, y_train, y_test, rf = irf_jupyter_utils.generate_rf_example(n_estimators=20, 
                                                                             feature_weight=None)
Explanation: Step 1: Fit the Initial Random Forest
Just fit every feature with equal weights per the usual random forest code e.g. DecisionForestClassifier in scikit-learn
End of explanation
print("Training feature dimensions", X_train.shape, sep = ":\n")
print("\n")
print("Training outcome dimensions", y_train.shape, sep = ":\n")
print("\n")
print("Test feature dimensions", X_test.shape, sep = ":\n")
print("\n")
print("Test outcome dimensions", y_test.shape, sep = ":\n")
print("\n")
print("first 2 rows of the training set features", X_train[:2], sep = ":\n")
print("\n")
print("first 2 rows of the training set outcomes", y_train[:2], sep = ":\n")
Explanation: Check out the data
End of explanation
all_rf_tree_data = irf_utils.get_rf_tree_data(
    rf=rf, X_train=X_train, X_test=X_test, y_test=y_test)
Explanation: Step 2: Get all Random Forest and Decision Tree Data
Extract in a single dictionary the random forest data and for all of it's decision trees
This is as required for RIT purposes
End of explanation
np.random.seed(12)
all_rit_tree_data = irf_utils.get_rit_tree_data(
    all_rf_tree_data=all_rf_tree_data,
    bin_class_type=1,
    M=100,
    max_depth=2,
    noisy_split=False,
    num_splits=2)
Explanation: STEP 3: Get the RIT data and produce RITs
End of explanation
# Print the feature ranking
print("Feature ranking:")
feature_importances_rank_idx = all_rf_tree_data['feature_importances_rank_idx']
feature_importances = all_rf_tree_data['feature_importances']
for f in range(X_train.shape[1]):
    print("%d. feature %d (%f)" % (f + 1
                                   , feature_importances_rank_idx[f]
                                   , feature_importances[feature_importances_rank_idx[f]]))
Explanation: Perform Manual CHECKS on the irf_utils
These should be converted to unit tests and checked with nosetests -v test_irf_utils.py
Step 4: Plot some Data
List Ranked Feature Importances
End of explanation
# Plot the feature importances of the forest
feature_importances_std = all_rf_tree_data['feature_importances_std']
plt.figure()
plt.title("Feature importances")
plt.bar(range(X_train.shape[1])
        , feature_importances[feature_importances_rank_idx]
        , color="r"
        , yerr = feature_importances_std[feature_importances_rank_idx], align="center")
plt.xticks(range(X_train.shape[1]), feature_importances_rank_idx)
plt.xlim([-1, X_train.shape[1]])
plt.show()
Explanation: Plot Ranked Feature Importances
End of explanation
# Now plot the trees individually
#irf_jupyter_utils.draw_tree(decision_tree = all_rf_tree_data['rf_obj'].estimators_[0])
Explanation: Decision Tree 0 (First) - Get output
Check the output against the decision tree graph
End of explanation
#irf_jupyter_utils.pretty_print_dict(inp_dict = all_rf_tree_data['dtree0'])
# Count the number of samples passing through the leaf nodes
sum(all_rf_tree_data['dtree0']['tot_leaf_node_values'])
Explanation: Compare to our dict of extracted data from the tree
End of explanation
#irf_jupyter_utils.pretty_print_dict(inp_dict = all_rf_tree_data['dtree0']['all_leaf_paths_features'])
Explanation: Check output against the diagram
End of explanation
all_rf_weights, all_K_iter_rf_data, \
all_rf_bootstrap_output, all_rit_bootstrap_output, \
stability_score = irf_utils.run_iRF(X_train=X_train,
                                    X_test=X_test,
                                    y_train=y_train,
                                    y_test=y_test,
                                    K=5,
                                    n_estimators=20,
                                    B=30,
                                    random_state_classifier=2018,
                                    propn_n_samples=.2,
                                    bin_class_type=1,
                                    M=20,
                                    max_depth=5,
                                    noisy_split=False,
                                    num_splits=2,
                                    n_estimators_bootstrap=5)
Explanation: Run the iRF function
We will run the iRF with the following parameters
Data:
breast cancer binary classification data
random state (for reproducibility): 2018
Weighted RFs
K: 5 iterations
number of trees: 20
Bootstrap RFs
proportion of bootstrap samples: 20%
B: 30 bootstrap samples
number of trees (bootstrap RFs): 5 iterations
RITs (on the bootstrap RFs)
M: 20 RITs per forest
filter label type: 1-class only
Max Depth: 5
Noisy Split: False
Number of splits at Node: 2 splits
Running the iRF is easy - single function call
All of the bootstrap, RIT complexity is covered through the key parameters passed through
in the main algorithm (as listed above)
This function call returns the following data:
all RF weights
all the K RFs that are iterated over
all of the B bootstrap RFs that are run
all the B*M RITs that are run on the bootstrap RFs
the stability score
This is a lot of data returned!
Will be useful when we build the interface later
Let's run it!
End of explanation
irf_utils._get_histogram(stability_score, sort = True)
Explanation: Examine the stability scores
End of explanation
for k in range(5): 
    
    iteration = "rf_iter{}".format(k)
    
    feature_importances_std = all_K_iter_rf_data[iteration]['feature_importances_std']
    feature_importances_rank_idx = all_K_iter_rf_data[iteration]['feature_importances_rank_idx']
    feature_importances = all_K_iter_rf_data[iteration]['feature_importances']
    plt.figure(figsize=(8, 6))
    title = "Feature importances; iteration = {}".format(k)
    plt.title(title)
    plt.bar(range(X_train.shape[1])
            , feature_importances[feature_importances_rank_idx]
            , color="r"
            , yerr = feature_importances_std[feature_importances_rank_idx], align="center")
    plt.xticks(range(X_train.shape[1]), feature_importances_rank_idx, rotation='vertical')
    plt.xlim([-1, X_train.shape[1]])
    plt.show()
    
    
    
Explanation: That's interesting - feature 22, 27, 20, 23 keep popping up!
We should probably look at the feature importances to understand if there is a useful correlation
Examine feature importances
In particular, let us see how they change over the K iterations of random forest
End of explanation
irf_jupyter_utils.pretty_print_dict(all_K_iter_rf_data['rf_iter4']['rf_validation_metrics'])
# Now plot the trees individually
irf_jupyter_utils.draw_tree(decision_tree = all_K_iter_rf_data['rf_iter4']['rf_obj'].estimators_[0])
Explanation: Some Observations
Note that after 5 iterations, the most important features were found to be 22, 27, 7, and 23
Now also recall that the most stable interactions were found to be '22_27', '7_22', '7_22_27', '23_27', '7_27', '22_23_27'
Given the overlap between these two plots, the results are not unreasonable here. 
Explore iRF Data Further
We can look at the decision paths of the Kth RF
Let's look at the final iteration RF - the key validation metrics
End of explanation
irf_jupyter_utils.pretty_print_dict(
    all_K_iter_rf_data['rf_iter4']['dtree0']['all_leaf_paths_features'])
Explanation: We can get this data quite easily in a convenient format
End of explanation
irf_jupyter_utils.pretty_print_dict(
    all_K_iter_rf_data['rf_iter4']['dtree0']['all_leaf_node_values'])
Explanation: This checks nicely against the plotted diagram above.
In fact - we can go further and plot some interesting data from the Decision Trees
- This can help us understand variable interactions better
End of explanation
irf_utils._hist_features(all_K_iter_rf_data['rf_iter4'], n_estimators = 20, \
                         title = 'Frequency of features along decision paths : iteration = 4')
Explanation: We can also look at the frequency that a feature appears along a decision path
End of explanation
all_K_iter_rf_data.keys()
print(all_K_iter_rf_data['rf_iter0']['feature_importances'])
Explanation: The most common features that appeared were 27,22,23, and 7. This matches well with the feature importance plot above. 
Run some Sanity Checks
Run iRF for just 1 iteration - should be the uniform sampling version
This is just a sanity check: the feature importances from iRF after 1 iteration should match the feature importance from running a standard RF
End of explanation
rf = RandomForestClassifier(n_estimators=20, random_state=2018)
rf.fit(X=X_train, y=y_train)
print(rf.feature_importances_)
Explanation: Compare to the original single fitted random forest
End of explanation
<END_TASK>