<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Decision trees and Random forest
Step1:  Data preprocessing
Step2:  Regression tree
Step3:  Randomly defined train and test set
Step4:  Know, we want to define the max_depht parameter that minimizes the out of sample using cross validation.
Step5:  Ploting tree
Step6:  Random Forest
Step7:  The random forest model has a lot of parameter we can optimize to get a better fit in our model.
- n_estimators = number of trees in the foreset
- max_depth = max number of levels in each decision tree
- min_samples_split = min number of data points placed in a node before the node is split
- min_samples_leaf = min number of data points allowed in a leaf node
Step8:  Even if with the random forest model we loss a lot of the model interpretation that a simple decision tree gave us, we can still get the importance of each feauture
Step9:  GBM
Some information about GBM'S parameters
Step10:  XGBOOST
some information on XGBOOST
Step11:  Classification problem
To see how we can apply decision trees to a classification problem we are going use the following data, to build models which objective is to predict if a person is diabetic.
Step12:  Optimize max_depth in classification problem
Step13:  Random forest for classification problems
<ASSISTANT_TASK:>
Python Code:
import pandas as pd
import numpy as np
import graphviz 
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeRegressor
from sklearn.metrics import mean_squared_error as mse
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import GridSearchCV
from tqdm import tqdm
import matplotlib.pyplot as plt
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score
from sklearn.ensemble import RandomForestRegressor
from sklearn import tree
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.tree import DecisionTreeClassifier
from xgboost import XGBClassifier
df = pd.read_csv('co_properties.csv.gz', compression='gzip', header=0,    sep=',', quotechar='"', error_bad_lines=False)
Explanation: Decision trees and Random forest
End of explanation
def categoricas(df,lista):
    dummies= pd.get_dummies(df[lista])
    df=df.drop(columns=lista)
    data = pd.concat([df, dummies], axis=1)
    return data
    
df=df[df['operation_type']=='Venta']
df=df[df['property_type']=='Casa']
df=df[df['currency']=='COP']
df=df[df['l3']=='Bogotá D.C']
df=df.drop(columns=['l6','id','ad_type','start_date','created_on','l1','l2','l3','end_date','title','price_period','title','description','property_type','operation_type','currency'])
df=df.dropna(subset=['l4','l5','price'])
df=categoricas(df,['l4','l5'])
variables=['lat', 'lon', 'rooms', 'bedrooms', 'bathrooms', 'surface_total',
       'surface_covered']
for i in variables:
    df.loc[df[i].isnull()==True,i+'null']=1
    df.loc[df[i].isnull()==False,i+'null']=0
    df.loc[df[i].isnull()==True,i]=-1
    
Explanation: Data preprocessing
End of explanation
data=df
df.columns
Explanation: Regression tree
End of explanation
x_train, x_test, y_train, y_test = train_test_split(data.drop(columns=['price']),data['price'], test_size=0.30,
                                                    random_state=200,
                                                    shuffle=True)
tree2=DecisionTreeRegressor().fit(x_train,y_train)
mse(y_test, tree2.predict(x_test))
Explanation: Randomly defined train and test set
End of explanation
model = DecisionTreeRegressor()
gs = GridSearchCV(model,
                  param_grid = {'max_depth': range(1, 30)},
                  cv=10,
                  n_jobs=10,
                  scoring='neg_mean_squared_error')
cv_tree1=gs.fit(x_train, y_train)
gs.best_estimator_
mse(y_test, cv_tree1.predict(x_test))
Explanation: Know, we want to define the max_depht parameter that minimizes the out of sample using cross validation.
End of explanation
from sklearn import tree
model=DecisionTreeRegressor(max_depth=3).fit(x_train,y_train)
data.columns
fn=data.columns
fig, axes = plt.subplots(nrows = 1,ncols = 1,figsize = (4,4), dpi=300)
tree.plot_tree(model,
               feature_names = fn, 
               filled = True)
fig.savefig('regression_tree.png')
Explanation: Ploting tree
End of explanation
rf = RandomForestRegressor(random_state = 42).fit(x_train,y_train)
mse(y_test, rf.predict(x_test))
Explanation: Random Forest
End of explanation
n_estimators =  [50,100,150]
max_depth = [10,20,30,40]
min_samples_split = [2, 5, 10]
min_samples_leaf = [1, 2, 4]
param_grid = {'n_estimators': n_estimators,
               'max_depth': max_depth,
               'min_samples_split': min_samples_split,
               'min_samples_leaf': min_samples_leaf,
              }
rf = RandomForestRegressor()
grid_search = GridSearchCV(estimator = rf, param_grid = param_grid, 
                          cv = 5, n_jobs = 10, verbose = 2)
rf_cv=grid_search.fit(x_train, y_train)
grid_search.best_estimator_
Explanation: The random forest model has a lot of parameter we can optimize to get a better fit in our model.
- n_estimators = number of trees in the foreset
- max_depth = max number of levels in each decision tree
- min_samples_split = min number of data points placed in a node before the node is split
- min_samples_leaf = min number of data points allowed in a leaf node
End of explanation
importances1 =rf_cv.best_estimator_.feature_importances_
importances_df1=pd.DataFrame({'importances':importances1,'feauture':data.drop(columns=['price']).columns})
importances_df1=importances_df1.sort_values(by=['importances'],ascending=False)
importances_df1
fig = plt.figure(figsize = (10, 5))
plt.bar(importances_df1.feauture[:15],importances_df1.importances[:15], color ='maroon',
        width = 0.4)
plt.xticks( rotation='vertical')
Explanation: Even if with the random forest model we loss a lot of the model interpretation that a simple decision tree gave us, we can still get the importance of each feauture
End of explanation
n_estimators =  [700]
max_depth = [10,20,30,40]
min_samples_split = [2, 5, 10]
learning_rate=[0.15,0.05,0.01,0.005]
param_grid = {'n_estimators': n_estimators,
               'max_depth': max_depth,
               'min_samples_split': min_samples_split,
              'learning_rate':learning_rate
              }
gbm= GradientBoostingRegressor()
grid_search = GridSearchCV(estimator = gbm, param_grid = param_grid, 
                          cv = 5, n_jobs = 7, verbose = 2,scoring='neg_mean_squared_error')
gbm_cv=grid_search.fit(x_train, y_train)
grid_search.best_estimator_
mse(y_test, gbm_cv.predict(x_test))
Explanation: GBM
Some information about GBM'S parameters
End of explanation
grid = {
        'learning_rate': [0.01, 0.1,0.2,0.3,0.5],
        'max_depth': [10,20,30,40,50],
        'objective': ['reg:squarederror']
    }
xgb_model = XGBRegressor()
gsearch = GridSearchCV(estimator = xgb_model,
                           param_grid = grid,                         
                           scoring = 'neg_mean_squared_error', 
                           cv = 5,
                           n_jobs = 7,
                           verbose = 1)
xg_cv=gsearch.fit(x_train,y_train)
mse(y_test, xg_cv.predict(x_test))
xg_cv.best_estimator_
df2= pd.Series(xg_cv.best_estimator_.feature_importances_, list(data.drop(columns=['price']))).sort_values(ascending=False)
df2[:15].plot(kind='bar', title='Importance of Features')
plt.ylabel('Feature Importance Score')
Explanation: XGBOOST
some information on XGBOOST
End of explanation
data2=pd.read_csv('diabetes.csv')
data2
x1_train, x1_test, y1_train, y1_test = train_test_split(data2.drop(columns=['Outcome']),data2['Outcome'], test_size=0.10,
                                                    random_state=200,
                                                    shuffle=True)
data2.isnull().sum()
clf1=DecisionTreeClassifier(max_depth=3).fit(x1_train,y1_train)
y_pred=clf1.predict(x1_test)
accuracy_score(y1_test, y_pred)
accuracy_score(y1_test, y_pred)
Explanation: Classification problem
To see how we can apply decision trees to a classification problem we are going use the following data, to build models which objective is to predict if a person is diabetic.
End of explanation
model = DecisionTreeClassifier()
gs = GridSearchCV(model,
                  param_grid = {'max_depth': range(1, 30)},
                  cv=10,
                  n_jobs=10,
                  scoring='accuracy')
clf2=gs.fit(x1_train, y1_train)
gs.best_estimator_
y_pred1=clf2.predict(x1_test)
accuracy_score(y1_test, y_pred1)
Explanation: Optimize max_depth in classification problem
End of explanation
rf_clf=RandomForestClassifier().fit(x1_train,y1_train)
y_pred2=rf_clf.predict(x1_test)
accuracy_score(y1_test, y_pred2)
rfcl = RandomForestClassifier()
n_estimators =  [50,100,150]
max_depth = [10,20,30,40]
min_samples_split = [2, 5, 10]
min_samples_leaf = [1, 2, 4]
param_grid = {'n_estimators': n_estimators,
               'max_depth': max_depth,
               'min_samples_split': min_samples_split,
               'min_samples_leaf': min_samples_leaf,
              }
grid_search = GridSearchCV(estimator = rfcl, param_grid = param_grid, 
                          cv = 5, n_jobs = 10, verbose = 2)
rfcl_cv=grid_search.fit(x1_train, y1_train)
y_pred3=rfcl_cv.predict(x1_test)
accuracy_score(y1_test, y_pred3)
importances =rfcl_cv.best_estimator_.feature_importances_
importances_df=pd.DataFrame({'importances':importances,'feauture':data2.drop(columns=['Outcome']).columns})
importances_df
fig = plt.figure(figsize = (10, 5))
plt.bar(importances_df.feauture,importances_df.importances, color ='maroon',
        width = 0.4)
plt.xticks( rotation='vertical')
Explanation: Random forest for classification problems
End of explanation
<END_TASK>