<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Drop some features
that might cause data leakage
that we don't have access to as inputs
Step1:  Note
Step2:  Prune out some more features
Step3:  Split data into training and test sets
Step4:  Random forest regressor
Step5:  Plot predicted values against labels
Step6:  It looks like it is overfitting
Plot model residuals
Step7:  All features show to be significant
Check the rank of the feature matrix
<ASSISTANT_TASK:>
Python Code:
train.columns
Explanation: Drop some features
that might cause data leakage
that we don't have access to as inputs
End of explanation
train.drop(['type', 
            'mv', 
            'blockTime', 
            'difficulty', 
            'gasLimit_b', 
            'gasUsed_b',
            'reward',
            'size',
            'totalFee',
            'gasShare',
            'gweiPaid',
            'gweiPaid_b',
            'gweiShare',
            'txcnt_second'], axis=1, inplace=True)
train.columns
Explanation: Note: there is an API that estimates the amount of gas used, thus we have access to gasUsed_t
End of explanation
train.drop(['free_t',
           'newContract',
           'amount_gwei',
           'type_enc',
           'dayofweek',
           'day'], axis=1, inplace=True)
Explanation: Prune out some more features
End of explanation
X = train.values
y = y_label
X_train, X_test, y_train, y_test = train_test_split(X, y)
X_train.shape, y_train.shape
Explanation: Split data into training and test sets
End of explanation
def rf_regressor(X_train, X_test, y_train, y_test):
    rf = RandomForestRegressor()
    rf.fit(X_train, y_train)
    y_pred = rf.predict(X_test)
    scores = cross_val_score(rf, X_train, y_train, scoring='r2', cv=5)
    print('MSE: {}'.format(mean_squared_error(y_test, y_pred)))
    print('R2_score: {}'.format(r2_score(y_test, y_pred)))
    print('avg_CV_score: {}'.format(np.mean(scores)))
    # write predicted values to csv
    p = pd.DataFrame({'y_pred': y_pred})
    p.to_csv('./../data/label_pred.csv')
    return rf
model = rf_regressor(X_train, X_test, y_train, y_test)
def plot_feature_importance(rf, feature_df):
    cols = []
    for col in feature_df.columns:
        cols.append(col)
    feat_scores = pd.DataFrame({'Fraction of Samples Affected' : rf.feature_importances_},
                           index=cols)
    feat_scores = feat_scores.sort_values(by='Fraction of Samples Affected')
    feat_scores.plot(kind='barh', color='r', figsize=(6,6))
    #plt.xlabel('Importance', fontsize=18)
    plt.title('Feature Importance', fontsize=18)
    plt.tight_layout()
    plt.savefig('./../images/feat_import_pruned.png', dpi=300)
plot_feature_importance(model, train)
Explanation: Random forest regressor
End of explanation
y_pred = pd.read_csv('./../data/label_pred.csv')
y_pred.drop('Unnamed: 0', axis=1, inplace=True)
y_pred.head()
y_test.shape
y_pred = y_pred.values.ravel()
y_test.shape, y_pred.shape
result = pd.DataFrame({'y_test': y_test, 'y_pred': y_pred})
result.head()
plt.scatter(result['y_test'], result['y_pred'])
plt.xlim(0,100)
plt.ylim(0,100)
sns.set(style="darkgrid", color_codes=True)
sns.jointplot(x="y_test", y="y_pred", data=result)
plt.xlim(0, 100)
plt.tight_layout()
plt.savefig('./../images/jointplot.png', dpi=300)
Explanation: Plot predicted values against labels
End of explanation
sns.residplot(result.y_test, result.y_pred , lowess=True, color="g")
def linear_regression(X_train, X_test, y_train, y_test):
    lr = LinearRegression()
    lr.fit(X_train, y_train)
    y_pred = lr.predict(X_test)
    scores = cross_val_score(lr, X_train, y_train, scoring='r2', cv=5)
    print('MSE: {}'.format(mean_squared_error(y_test, y_pred)))
    print('R2_score: {}'.format(r2_score(y_test, y_pred)))
    print('avg_CV_score: {}'.format(np.mean(scores)))
    return lr
linear_regression(X_train, X_test, y_train, y_test)
# get summary statistics from statsmodels
model = sm.OLS(y_train, X_train)
result = model.fit()
result.summary()
Explanation: It looks like it is overfitting
Plot model residuals
End of explanation
from numpy.linalg import matrix_rank
X_train.shape
matrix_rank(X_train)
Explanation: All features show to be significant
Check the rank of the feature matrix
End of explanation
<END_TASK>