<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 DECISION TREE
is another method we can use for finding relationship between target and one or more predictors
decision trees can be used for both categorical and continous targets (so both classification and regression tasks -  today we'll be focusing on regression trees)
the main idea is to create a tree of decisions that best partitions the data 
creating a tree involves deciding on which features to split the data and what conditions to use for splitting, as well as with knowing when to stop
Step1:  RANDOM FORESTS
are using Bagging (bootstrap aggregating) algorithm
bagging is an ensemble learning where we build each model using the same algorithm, but we train each learner on different set of data
GRADIENT BOOSTED TREES
are using Boosting (Ada Boost) algorithm
boosting is a variation of ensemble trees that strives to improve the learners by focusing on areas where the system is not performing well.
We can start by creating one decision tree of depth 3 using all features
Step2:  The first avalanche problem dictates the danger level - that was expected 
Step3:  Looks like there is little gain when using a depth > 7.
Step4:  Now we see slight improvement in $R^{2}$.
Maybe we'll get better results with random forests and/or gradient boosted trees. 
Random forest
Step5:  Gradient boosting
Step6:  Again, very small difference between random forests and boosted trees.
Last thing we can check is imprtance of variables - if some of the feature are not as useful as the other in explaining the variability in our target variable, we cn exclude them in order to simplify our model.
<ASSISTANT_TASK:>
Python Code:
import pandas as pd
import numpy as np
import json
import graphviz
import matplotlib.pyplot as plt
from sklearn import tree
from sklearn.model_selection import train_test_split
pd.set_option("display.max_rows",6)
%matplotlib inline
df_data = pd.read_csv(r'varsom_ml_preproc_3y.csv', index_col=0)
target_ = 'danger_level'
X = df_data.drop([target_, 'date'], axis=1)
y = df_data.filter([target_], axis=1)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state = 222, test_size = 0.3)
print(df_data.columns.values)
df_data.describe()
print(df_data[df_data['danger_level']>3]['region_id'].unique())
Explanation: DECISION TREE
is another method we can use for finding relationship between target and one or more predictors
decision trees can be used for both categorical and continous targets (so both classification and regression tasks -  today we'll be focusing on regression trees)
the main idea is to create a tree of decisions that best partitions the data 
creating a tree involves deciding on which features to split the data and what conditions to use for splitting, as well as with knowing when to stop
End of explanation
dec_tree = tree.DecisionTreeRegressor(random_state=222, max_depth = 3)
dec_tree.fit(X_train, y_train) # we're using the same data as in last linear model
predictions_dt = dec_tree.predict(X_test)
print(predictions_dt.shape, y_test.shape)
# Visualize the tree
dot_data = tree.export_graphviz(dec_tree, out_file=None, 
                         feature_names=df_data.drop([target_, 'date'], axis=1).columns,   
                         filled=True, rounded=True,  
                         special_characters=True)  
graph = graphviz.Source(dot_data) 
graph.render('aval_danger_by_problem_dt_lev7')
graph 
Explanation: RANDOM FORESTS
are using Bagging (bootstrap aggregating) algorithm
bagging is an ensemble learning where we build each model using the same algorithm, but we train each learner on different set of data
GRADIENT BOOSTED TREES
are using Boosting (Ada Boost) algorithm
boosting is a variation of ensemble trees that strives to improve the learners by focusing on areas where the system is not performing well.
We can start by creating one decision tree of depth 3 using all features
End of explanation
print('Decision tree R^2: %.4f' % dec_tree.score(X_test, y_test)) 
depths = range(1, 20)
tree_models = [tree.DecisionTreeRegressor(random_state=222, max_depth=d).fit(X_train, y_train) for d in depths]
tree_Rsquare = [f.score(X_test, y_test) for f in tree_models]
plt.plot(depths, tree_Rsquare, color = 'red')
plt.xlabel('Tree depth')
plt.ylabel('$R^2$')
Explanation: The first avalanche problem dictates the danger level - that was expected :-)
We need to evaluate our model:
End of explanation
# so let's create a tree with depth = 7
dec_tree = tree.DecisionTreeRegressor(random_state=222, max_depth = 7)
dec_tree.fit(X_train, y_train) # we're using the same data as in last linear model
predictions_dt = dec_tree.predict(X_test)
# Visualize the tree
dot_data = tree.export_graphviz(dec_tree, out_file=None, 
                         feature_names=df_data.drop([target_, 'date'], axis=1).columns,   
                         filled=True, rounded=True,  
                         special_characters=True)  
graph = graphviz.Source(dot_data) 
graph.render('aval_danger_by_problem_dt2')
graph 
print('Decision tree R^2: %.4f' % dec_tree.score(X_test, y_test)) 
Explanation: Looks like there is little gain when using a depth > 7.
End of explanation
from sklearn.ensemble import RandomForestRegressor
forest_reg = RandomForestRegressor(random_state = 422, max_depth = 7)
forest_reg.fit(X_train, y_train)
predictions_rf = forest_reg.predict(X_test)
print('Random forest R^2: %.4f' % forest_reg.score(X_test, y_test)) 
Explanation: Now we see slight improvement in $R^{2}$.
Maybe we'll get better results with random forests and/or gradient boosted trees. 
Random forest
End of explanation
from sklearn.ensemble import GradientBoostingRegressor
grad_boost = GradientBoostingRegressor(random_state = 222, max_depth = 6)
grad_boost.fit(X_train, y_train)
predictions_gb = grad_boost.predict(X_test)
print('Gradient boosted tree R^2: %.4f' % grad_boost.score(X_test, y_test)) 
Explanation: Gradient boosting
End of explanation
feature_labels = np.array(df_data.drop([target_, 'date'], axis=1).columns)
#feature_labels = np.array(['CRIM', 'ZN', 'INDUS', 'CHAS', 'RM', 'AGE', 'RAD', 'TAX', 'PTRATIO', 'B', 'LSTAT', 'MEDV'])
importance = forest_reg.feature_importances_
feature_indexes_by_importance = importance.argsort()
for index in feature_indexes_by_importance:
    print('{}-{:.2f}%'.format(feature_labels[index], (importance[index] *100.0)))
# excluding the variables where impotance is less than 1%
X = df_data[['avalanche_problem_1_distribution_id',
             'avalanche_problem_1_destructive_size_ext_id',
             'avalanche_problem_1_trigger_simple_id',
             'avalanche_problem_1_probability_id',
             'avalanche_problem_2_destructive_size_ext_id',
             'avalanche_problem_2_probability_id']]
y = df_data['danger_level']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.3, random_state = 222)
grad_boost = GradientBoostingRegressor(random_state = 222, max_depth = 7)
grad_boost.fit(X_train, y_train)
predictions_gb = grad_boost.predict(X_test)
print('Gradient boosted tree R^2: %.4f' % grad_boost.score(X_test, y_test)) 
Explanation: Again, very small difference between random forests and boosted trees.
Last thing we can check is imprtance of variables - if some of the feature are not as useful as the other in explaining the variability in our target variable, we cn exclude them in order to simplify our model.
End of explanation
<END_TASK>