<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Random Forest
In random forests, each tree in the ensemble is built from a sample drawn with replacement (i.e., a bootstrap sample) from the training set. In addition, when splitting a node during the construction of the tree, the split that is chosen is no longer the best split among all features. Instead, the split that is picked is the best split among a random subset of the features. 
As a result of this randomness, the bias of the forest usually slightly increases (with respect to the bias of a single non-random tree) but, due to averaging, its variance also decreases, usually more than compensating for the increase in bias, hence yielding an overall better model.
Data Preparation
Step1:  Implementing Random Forest
Step2:  Key input parameters (in addition to decision trees)
bootstrap
Step3:  Out-of-Bag Error
The out-of-bag (OOB) error is the average error for each training observation calculated using predictions from the trees that do not contain it in their respective bootstrap sample. This allows the RandomForest to be fit and validated whilst being trained.
<ASSISTANT_TASK:>
Python Code:
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
plt.style.use('ggplot')
train = pd.read_csv("data/trainRF.csv")
test = pd.read_csv("data/testRF.csv")
train_13 = train.drop(['day','month', 'duration'], axis = 1)
test_13 = test.drop(['day','month', 'duration'], axis = 1)
from sklearn import preprocessing
categorical_variables =train_13.dtypes[train_13.dtypes=="object"].index.tolist()
for i in categorical_variables:
    lbl = preprocessing.LabelEncoder()
    lbl.fit(list(train_13[i]))
    train_13[i] = lbl.transform(train_13[i])
    test_13[i] = lbl.transform(test_13[i])
train_13_data = train_13.ix[:,0:13]
train_13_target = train_13.iloc[:, -1]
test_13_data = train_13.ix[:,0:13]
test_13_target = train_13.iloc[:, -1]
Explanation: Random Forest
In random forests, each tree in the ensemble is built from a sample drawn with replacement (i.e., a bootstrap sample) from the training set. In addition, when splitting a node during the construction of the tree, the split that is chosen is no longer the best split among all features. Instead, the split that is picked is the best split among a random subset of the features. 
As a result of this randomness, the bias of the forest usually slightly increases (with respect to the bias of a single non-random tree) but, due to averaging, its variance also decreases, usually more than compensating for the increase in bias, hence yielding an overall better model.
Data Preparation
End of explanation
from sklearn.ensemble import RandomForestClassifier
clf = RandomForestClassifier()
clf.fit(train_13_data, train_13_target)
Explanation: Implementing Random Forest
End of explanation
importances = clf.feature_importances_
# Importance of the features in the forest
importances
#Calculate the standard deviation of variable importance
std = np.std([tree.feature_importances_ for tree in clf.estimators_], axis=0)
std
indices = np.argsort(importances)[::-1]
indices
length = train_13_data.shape[1]
labels = []
for i in range(length):
    labels.append(train_13_data.columns[indices[i]])
# Plot the feature importances of the forest
plt.figure(figsize=(16, 6))
plt.title("Feature importances")
plt.bar(range(length), importances[indices], yerr=std[indices], align="center")
plt.xticks(range(length), labels)
plt.xlim([-1, length])
plt.show()
Explanation: Key input parameters (in addition to decision trees)
bootstrap: Whether bootstrap samples are used when building trees
max_features: The number of features to consider when looking for the best split (auto = sqrt) 
n_estimators: The number of trees in the forest
oob_score: Whether to use out-of-bag samples to estimate the generalization accuracy
Key output parameters
Feature Importance: The higher, the more important the feature
Out-of-Bag Score: Validation score of the training dataset obtained using an out-of-bag estimate.
Feature Importance
The relative rank (i.e. depth) of a feature used as a decision node in a tree can be used to assess the relative importance of that feature with respect to the predictability of the target variable. Features used at the top of the tree contribute to the final prediction decision of a larger fraction of the input samples. The expected fraction of the samples they contribute to can thus be used as an estimate of the relative importance of the features.
By averaging those expected activity rates over random trees one can reduce the variance of such an estimate and use it for feature selection.
End of explanation
import warnings
warnings.filterwarnings('ignore')
clf2 = RandomForestClassifier(warm_start=True, oob_score=True, max_features=None)
clf2.fit(train_13_data, train_13_target)
clf2.oob_score_
min_estimators = 10
max_estimators = 50
error_rate = []
for i in range(min_estimators, max_estimators + 1):
    clf2.set_params(n_estimators=i)
    clf2.fit(train_13_data, train_13_target)
    oob_error = 1 - clf2.oob_score_
    error_rate.append(oob_error)
error_rate_indice = [x for x in range(min_estimators, max_estimators + 1)]
plt.figure()
plt.plot(error_rate_indice, error_rate)
plt.xlim(min_estimators, max_estimators)
plt.xlabel("n_estimators")
plt.ylabel("OOB error rate")
plt.show()
Explanation: Out-of-Bag Error
The out-of-bag (OOB) error is the average error for each training observation calculated using predictions from the trees that do not contain it in their respective bootstrap sample. This allows the RandomForest to be fit and validated whilst being trained.
End of explanation
<END_TASK>