<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Shap XGBoost
Feature importance with SHAP
While XgBoost uses Gain to rank feature importance by default, and some more important features may rank lower, this algorithm is trying to deal with this drawback
Learning resources
Step1:  Summarize Feature Importance
Summary Plot
Higher feature value (pink) with longer length means higher value has larger impact to the prediction. Towards left means impact more on lower value/negative class, towards right means impoact more on higher value/positive class.
From upper to lower, features are ranked by importance decreasing order
Step2:  Check Individual Cases
It can only show 2 records here, the red one means higher feature value, the blue one means lower feature value. But none of them means it's high or low in the whole population.
When a record is at the top left, it means it tends to contribute to the negative class/lower value prediction; top right means it tends to contribute to the positive class/higher value prediction.
But just today, in the real work, we saw opposite cases, where real high probability positive class has feature values all indicate to negative class..
<ASSISTANT_TASK:>
Python Code:
import xgboost as xgb
import shap
from sklearn.model_selection import train_test_split
import pandas as pd
X,y = shap.datasets.boston()
X.head()
print(y.shape)  # predict house price
y[4:10]
y = pd.DataFrame(y)
y.head()
# for regression method, I can not use stratify split with this method
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=410, train_size=0.75, test_size=0.25)
X_train.reset_index(drop=True, inplace=True)
X_test.reset_index(drop=True, inplace=True)
y_train.reset_index(drop=True, inplace=True)
y_test.reset_index(drop=True, inplace=True)
param_dist = {'learning_rate':0.01}
model = xgb.XGBRegressor(**param_dist)
model.fit(X_train, y_train,
        eval_set=[(X_train, y_train), (X_test, y_test)],
        verbose=False)
explainer = shap.TreeExplainer(model)
shap_values = explainer.shap_values(X)
print(shap_values.shape)
shap_values
# If you JS load successfully, this will generate interactive visualization
shap.initjs()
check_row = 7  # check each individual case
shap.force_plot(explainer.expected_value, shap_values[check_row,:], X.iloc[check_row,:])
Explanation: Shap XGBoost
Feature importance with SHAP
While XgBoost uses Gain to rank feature importance by default, and some more important features may rank lower, this algorithm is trying to deal with this drawback
Learning resources: https://github.com/slundberg/shap/tree/master/notebooks
End of explanation
shap.summary_plot(shap_values, X)
X.iloc[7:9, :]
shap_values[7:9, :]
Explanation: Summarize Feature Importance
Summary Plot
Higher feature value (pink) with longer length means higher value has larger impact to the prediction. Towards left means impact more on lower value/negative class, towards right means impoact more on higher value/positive class.
From upper to lower, features are ranked by importance decreasing order
End of explanation
shap.summary_plot(shap_values[7:9, :], X.iloc[7:9, :])
# comparing with xgboost feature importance (by default it's using gain to rank fearure importance)
print(X.columns)
model.feature_importances_
# using absolute mean value of SHAP values to rank the features
shap.summary_plot(shap_values, X, plot_type="bar")
Explanation: Check Individual Cases
It can only show 2 records here, the red one means higher feature value, the blue one means lower feature value. But none of them means it's high or low in the whole population.
When a record is at the top left, it means it tends to contribute to the negative class/lower value prediction; top right means it tends to contribute to the positive class/higher value prediction.
But just today, in the real work, we saw opposite cases, where real high probability positive class has feature values all indicate to negative class..
End of explanation
<END_TASK>