<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Get the results of a single run
Step1:  Done. Let's test the reshape_by_symbol function
Step2:  So, the reshape_by_symbol function seems to work with run_single_val. It could be added to it. Let's test the roll_evaluate function.
Step3:  Let's do some previous filtering to avoid problems
<ASSISTANT_TASK:>
Python Code:
from predictor import evaluation as ev
from predictor.dummy_mean_predictor import DummyPredictor
predictor = DummyPredictor()
y_train_true_df, y_train_pred_df, y_val_true_df, y_val_pred_df = ev.run_single_val(x, y, ahead_days, predictor)
print(y_train_true_df.shape)
print(y_train_pred_df.shape)
print(y_val_true_df.shape)
print(y_val_pred_df.shape)
y_train_true_df.head()
y_train_pred_df.head()
y_val_true_df.head()
y_val_pred_df.head()
Explanation: Get the results of a single run
End of explanation
y_train_true_rs = ev.reshape_by_symbol(y_train_true_df)
print(y_train_true_rs.shape)
y_train_true_rs.head()
y_train_pred_rs = ev.reshape_by_symbol(y_train_pred_df)
print(y_train_pred_rs.shape)
y_train_pred_rs.head()
y_val_true_rs = ev.reshape_by_symbol(y_val_true_df)
print(y_val_true_rs.shape)
y_val_true_rs.head()
Explanation: Done. Let's test the reshape_by_symbol function
End of explanation
u = x.index.levels[0][0]
print(u)
fe.SPY_DF.sort_index().index.unique()
md = fe.SPY_DF.index.unique()
u in md
fe.add_market_days(u,6)
Explanation: So, the reshape_by_symbol function seems to work with run_single_val. It could be added to it. Let's test the roll_evaluate function.
End of explanation
# Getting the data
GOOD_DATA_RATIO = 0.99
data_df = pd.read_pickle('../../data/data_train_val_df.pkl')
sys.path.append('../../')
import predictor.feature_extraction as fe
import utils.preprocessing as pp
data_df = pp.drop_irrelevant_symbols(data_df, GOOD_DATA_RATIO)
train_time = -1 # In real time days
base_days = 7 # In market days
step_days = 7 # market days
ahead_days = 1 # market days
today = data_df.index[-1] # Real date
tic = time()
x, y = fe.generate_train_intervals(data_df, 
                                   train_time, 
                                   base_days, 
                                   step_days, 
                                   ahead_days, 
                                   today, 
                                   fe.feature_close_one_to_one)
toc = time()
print('Elapsed time: %i seconds.' % (toc-tic))
print(data_df.shape)
data_df.head()
SAMPLES_GOOD_DATA_RATIO = 0.9
x_y_df = pd.concat([x, y], axis=1)
x_y_df = pp.drop_irrelevant_samples(x_y_df, SAMPLES_GOOD_DATA_RATIO)
x = x_y_df.iloc[:, :-1]
y = x_y_df.iloc[:, -1]
x = pp.fill_missing(x)
x_y_df.isnull().sum()
x.isnull().sum().sum()
y.isnull().sum()
x_reshaped = ev.reshape_by_symbol(x)
x_reshaped.head()
x_reshaped.isnull().sum().max()
x.shape
x_reshaped.shape
x_reshaped[x_reshaped.notnull()]
y_train_true_df, y_train_pred_df, y_val_true_df, y_val_pred_df = ev.run_single_val(x, y, ahead_days, predictor)
from sklearn.metrics import r2_score
r2_score(y_train_true_df, y_train_pred_df, multioutput='raw_values')
tickers = y_train_true_df.index.levels[1]
tickers
y_train_true_df.loc[(slice(None), 'AAPL'),:]
from sklearn.metrics import r2_score
r2_train_score = []
mre_train = []
for ticker in tickers:
    y_true = y_train_true_df.loc[(slice(None), ticker),:]
    y_pred = y_train_pred_df.loc[(slice(None), ticker),:]
    r2_train_score.append(r2_score(y_true, y_pred))
    mre_train.append(ev.mre(y_true, y_pred))
np.mean(r2_train_score)
np.mean(mre_train)
plt.plot(mre_train)
ev.get_metrics(y_train_true_df, y_train_pred_df)
train_days = 252
x_y_sorted = pd.concat([x, y], axis=1).sort_index()
start_date = x_y_sorted.index.levels[0][0]
end_date = fe.add_market_days(start_date, train_days)
start_date
end_date
start_date + ((end_date - start_date) / 2)
train_days = 252
step_eval_days = 30
r2, mre, y_val_true_df, y_val_pred_df, mean_dates = ev.roll_evaluate(x, 
                                                                     y, 
                                                                     train_days, 
                                                                     step_eval_days, 
                                                                     ahead_days, 
                                                                     predictor, 
                                                                     verbose=True)
print(r2.shape)
print(mre.shape)
print(y_val_true_df.shape)
print(y_val_pred_df.shape)
print(mean_dates.shape)
plt.plot(mean_dates, r2[:, 0], 'b', label='Mean r2 score')
plt.plot(mean_dates, r2[:, 0] + 2*r2[:, 1], 'r')
plt.plot(mean_dates, r2[:, 0] - 2*r2[:, 1], 'r')
plt.xlabel('Mean date of the training period')
plt.legend()
plt.grid()
plt.plot(mean_dates, mre[:, 0], 'b', label='Mean MRE')
plt.plot(mean_dates, mre[:, 0] + 2*mre[:, 1], 'r')
plt.plot(mean_dates, mre[:, 0] - 2*mre[:, 1], 'r')
plt.xlabel('Mean date of the training period')
plt.legend()
plt.grid()
y_val_true_df.head()
y_val_pred_df.head()
r2_scores, mre_scores, tickers = ev.get_metrics(y_val_true_df, y_val_pred_df)
eval_df = pd.DataFrame(np.array([r2_scores, mre_scores]).T, index=tickers, columns=['r2', 'mre'])
eval_df.head()
eval_df['mre'].plot()
eval_df['r2'].plot()
eval_df.sort_values(by='mre', ascending=False)
plt.scatter(eval_df['r2'], eval_df['mre'])
eval2_df = ev.get_metrics_df(y_val_true_df, y_val_pred_df)
eval2_df.head()
Explanation: Let's do some previous filtering to avoid problems
End of explanation
<END_TASK>