<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 That looks like the best way to represent the data if we want to calculate the $R^2$ distance on a per-symbol basis. I could add it to the single val function.
Step1:  Now, let's implement the rolling validation.
Step2:  So, I could use a training period based in an amount of market days, or in an amount of sample base periods. The first approach would be taking into consideration the temporal correlation of the data, the second would consider that the amount of samples should be large enough. Not to lose sight of the real problem at hand, I will use the market days approach, and then check that the amount of samples is big enough.
Step3:  A lot of attention should be paid to the effect of filling the missing data. It may change the whole results.
Step5:  That last number is an approximation of the number of train/evaluation sets that are being considered.
Step6:  Let's test the whole process
Step7:  It seems like the weird point, in which the model is predicting terribly may be the 2008 financial crisis. And the big unpredictability is limited to one symbol. I should implement a way to trace the symbols...
What about the mean absolute error?
<ASSISTANT_TASK:>
Python Code:
def run_single_val(x, y, ahead_days, estimator):
    multiindex = x.index.nlevels > 1
    
    x_y = pd.concat([x, y], axis=1)
    x_y_sorted = x_y.sort_index()
    if multiindex:
        x_y_train = x_y_sorted.loc[:fe.add_market_days(x_y_sorted.index.levels[0][-1], -ahead_days)]
        x_y_val = x_y_sorted.loc[x_y_sorted.index.levels[0][-1]:]
    else:
        x_y_train = x_y_sorted.loc[:fe.add_market_days(x_y_sorted.index[-1], -ahead_days)]
        x_y_val = x_y_sorted.loc[x_y_sorted.index[-1]:]
    
    x_train = x_y_train.iloc[:,:-1]
    x_val = x_y_val.iloc[:,:-1]
    y_train_true = x_y_train.iloc[:,-1]
    y_val_true = x_y_val.iloc[:,-1]
    
    estimator.fit(x_train)
    y_train_pred = estimator.predict(x_train)
    y_val_pred = estimator.predict(x_val)
    
    y_train_true_df = pd.DataFrame(y_train_true)
    y_train_pred_df = pd.DataFrame(y_train_pred)
    y_val_true_df = pd.DataFrame(y_val_true)
    y_val_pred_df = pd.DataFrame(y_val_pred)
    
    return y_train_true, \
           y_train_pred, \
           y_val_true, \
           y_val_pred
y_train_true, y_train_pred, y_val_true, y_val_pred = run_single_val(x, y, 1, predictor)
print(y_train_true.shape)
print(y_train_pred.shape)
print(y_val_true.shape)
print(y_val_pred.shape)
print(y_train_true.shape)
y_train_true.head()
y = y_train_true
multiindex = y.index.nlevels > 1
    
if multiindex:
    DATE_LEVEL_NAME = 'level_0'
else:
    DATE_LEVEL_NAME = 'index'
DATE_LEVEL_NAME
y.reset_index()
reshape_by_symbol(y_train_true)
Explanation: That looks like the best way to represent the data if we want to calculate the $R^2$ distance on a per-symbol basis. I could add it to the single val function.
End of explanation
train_eval_days = -1 # In market days
base_days = 7 # In market days
step_days = 7 # market days
ahead_days = 1 # market days
today = data_df.index[-1] # Real date
train_days = 252 # market days per training period
step_eval_days =  30 # market days between training periods beginings
filled_data_df = pp.fill_missing(data_df)
tic = time()
x, y = fe.generate_train_intervals(filled_data_df, 
                                   train_eval_days, 
                                   base_days, 
                                   step_days, 
                                   ahead_days, 
                                   today, 
                                   fe.feature_close_one_to_one)
toc = time()
print('Elapsed time: %i seconds.' % (toc-tic))
x_y_sorted = pd.concat([x, y], axis=1).sort_index()
x_y_sorted
start_date = x_y_sorted.index.levels[0][0]
start_date
end_date = fe.add_market_days(start_date, 252)
end_date
end_date = fe.add_index_days(start_date, 252, x_y_sorted)
end_date
Explanation: Now, let's implement the rolling validation.
End of explanation
end_date = fe.add_market_days(start_date, 252)
x_i = x_y_sorted.loc[start_date:end_date].iloc[:,:-1]
y_i = x_y_sorted.loc[start_date:end_date].iloc[:,-1]
print(x_i.shape)
print(x_i.head())
print(y_i.shape)
print(y_i.head())
ahead_days
predictor = dmp.DummyPredictor()
y_train_true, y_train_pred, y_val_true, y_val_pred = run_single_val(x_i, y_i, ahead_days, predictor)
print(y_train_true.shape)
print(y_train_pred.shape)
print(y_val_true.shape)
print(y_val_pred.shape)
y_train_pred.head()
y_train_pred.dropna(axis=1, how='all').shape
scores = r2_score(pp.fill_missing(y_train_pred), pp.fill_missing(y_train_true), multioutput='raw_values')
print('R^2 score = %f +/- %f' % (np.mean(scores), 2*np.std(scores)))
scores = r2_score(y_train_pred, y_train_true, multioutput='raw_values')
print('R^2 score = %f +/- %f' % (np.mean(scores), np.std(scores)))
len(scores)
y_val_true_df = pd.DataFrame()
y_val_true
y_val_true_df.append(y_val_true)
Explanation: So, I could use a training period based in an amount of market days, or in an amount of sample base periods. The first approach would be taking into consideration the temporal correlation of the data, the second would consider that the amount of samples should be large enough. Not to lose sight of the real problem at hand, I will use the market days approach, and then check that the amount of samples is big enough.
End of explanation
x.index.min()
x.index.max()
x.index.max() - x.index.min()
(x.index.max() - fe.add_market_days(x.index.min(), train_days)).days // step_days
Explanation: A lot of attention should be paid to the effect of filling the missing data. It may change the whole results.
End of explanation
def roll_evaluate(x, y, train_days, step_eval_days, ahead_days, verbose=False):
    Warning: The final date of the period should be no larger than the final date of the SPY_DF
    
    # calculate start and end date
    # sort by date
    x_y_sorted = pd.concat([x, y], axis=1).sort_index()
    start_date = x_y_sorted.index[0]
    end_date = fe.add_market_days(start_date, train_days)
    final_date = x_y_sorted.index[-1]
    
    # loop: run_single_val(x,y, ahead_days, estimator)
    r2_train_means = []
    r2_train_stds = []
    y_val_true_df = pd.DataFrame()
    y_val_pred_df = pd.DataFrame()
    num_training_sets = (252/365) * (x.index.max() - fe.add_market_days(x.index.min(), train_days)).days // step_eval_days
    set_index = 0
    if verbose:
        print('Evaluating approximately %i training/evaluation pairs' % num_training_sets)
    
    while end_date < final_date:
        x = x_y_sorted.loc[start_date:end_date].iloc[:,:-1]
        y = x_y_sorted.loc[start_date:end_date].iloc[:,-1]
        y_train_true, y_train_pred, y_val_true, y_val_pred = run_single_val(x, y, ahead_days, predictor)
        # Calculate R^2 for training and append
        scores = r2_score(y_train_true, y_train_pred, multioutput='raw_values')
        r2_train_means.append(np.mean(scores))
        r2_train_stds.append(np.std(scores))
        # Append validation results
        y_val_true_df = y_val_true_df.append(y_val_true)
        y_val_pred_df = y_val_pred_df.append(y_val_pred)
        
        # Update the dates
        start_date = fe.add_market_days(start_date, step_eval_days)
        end_date = fe.add_market_days(end_date, step_eval_days)
        
        set_index += 1
        if verbose:
            sys.stdout.write('\rApproximately %2.1f percent complete.    ' % (100.0 * set_index / num_training_sets))
            sys.stdout.flush()
    
    return r2_train_means, r2_train_stds, y_val_true_df, y_val_pred_df
Explanation: That last number is an approximation of the number of train/evaluation sets that are being considered.
End of explanation
train_eval_days = -1 # In market days
base_days = 14 # In market days
step_days = 30 # market days
ahead_days = 1 # market days
today = data_df.index[-1] # Real date
filled_data_df = pp.fill_missing(data_df)
tic = time()
x, y = fe.generate_train_intervals(filled_data_df, 
                                   train_eval_days, 
                                   base_days, 
                                   step_days, 
                                   ahead_days, 
                                   today, 
                                   fe.feature_close_one_to_one)
toc = time()
print('Elapsed time: %i seconds.' % (toc-tic))
train_days = 252 # market days per training period
step_eval_days =  10 # market days between training periods beginings
tic = time()
r2_train_means, r2_train_stds, y_val_true_df, y_val_pred_df = roll_evaluate(x, y, train_days, step_eval_days, ahead_days, verbose=True)
toc = time()
print('Elapsed time: %i seconds.' % (toc-tic))
y_val_true_df.head()
pd.DataFrame(r2_train_means).describe()
scores = r2_score(y_val_true_df.T, y_val_pred_df.T, multioutput='raw_values')
print('R^2 score = %f +/- %f' % (np.mean(scores), np.std(scores)))
pd.DataFrame(scores).describe()
plt.plot(y_val_true_df.index, r2_train_means, label='r2_train_means')
plt.plot(y_val_true_df.index, scores, label='r2 validation scores')
plt.legend(loc='lower left')
scores_val = r2_score(y_val_true_df, y_val_pred_df, multioutput='raw_values')
print('R^2 score = %f +/- %f' % (np.mean(scores_val), np.std(scores_val)))
plt.plot(scores_val, label='r2 validation scores')
sorted_means = x.sort_index().mean(axis=1)
sorted_means.head()
sorted_means.plot()
sub_period = sorted_means['2009-03-01':]
plt.scatter(sub_period.index, sub_period)
Explanation: Let's test the whole process
End of explanation
from sklearn.metrics import mean_absolute_error
scores = mean_absolute_error(y_val_true_df.T, y_val_pred_df.T, multioutput='raw_values')
print('MAE score = %f +/- %f' % (np.mean(scores), np.std(scores)))
plt.plot(y_val_true_df.index, scores, label='MAE validation scores')
plt.legend(loc='lower left')
pd.DataFrame(scores).describe()
scores = mean_absolute_error(y_val_true_df, y_val_pred_df, multioutput='raw_values')
print('MAE score = %f +/- %f' % (np.mean(scores), np.std(scores)))
plt.plot(scores, label='MAE validation scores')
plt.legend(loc='lower left')
pd.DataFrame(scores).describe()
Explanation: It seems like the weird point, in which the model is predicting terribly may be the 2008 financial crisis. And the big unpredictability is limited to one symbol. I should implement a way to trace the symbols...
What about the mean absolute error?
End of explanation
<END_TASK>