<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 In this notebook the datsets for the predictor will be generated.
Step1:  Let's first define the list of parameters to use in each dataset.
Step2:  Now, let's define the function to generate each dataset.
Note
<ASSISTANT_TASK:>
Python Code:
# Basic imports
import os
import pandas as pd
import matplotlib.pyplot as plt
import numpy as np
import datetime as dt
import scipy.optimize as spo
import sys
from time import time
from sklearn.metrics import r2_score, median_absolute_error
%matplotlib inline
%pylab inline
pylab.rcParams['figure.figsize'] = (20.0, 10.0)
%load_ext autoreload
%autoreload 2
sys.path.append('../../')
import predictor.feature_extraction as fe
import utils.preprocessing as pp
Explanation: In this notebook the datsets for the predictor will be generated.
End of explanation
# Input values
GOOD_DATA_RATIO = 0.99  # The ratio of non-missing values for a symbol to be considered good
SAMPLES_GOOD_DATA_RATIO = 0.9  # The ratio of non-missing values for an interval to be considered good
train_val_time = -1  # In real time days (-1 is for the full interval)
''' Step days will be fixed. That means that the datasets with longer base periods will have samples 
that are more correlated. '''
step_days = 7  # market days
base_days = [7, 14, 28, 56, 112]  # In market days
ahead_days = [1, 7, 14, 28, 56]  # market days
datasets_params_list_df = pd.DataFrame([(x,y) for x in base_days for y in ahead_days],
                                      columns=['base_days', 'ahead_days'])
datasets_params_list_df['train_val_time'] = train_val_time
datasets_params_list_df['step_days'] = step_days
datasets_params_list_df['GOOD_DATA_RATIO'] = GOOD_DATA_RATIO
datasets_params_list_df['SAMPLES_GOOD_DATA_RATIO'] = SAMPLES_GOOD_DATA_RATIO
datasets_params_list_df
Explanation: Let's first define the list of parameters to use in each dataset.
End of explanation
def generate_one_set(params):
    # print(('-'*70 + '\n {}, {} \n' + '-'*70).format(params['base_days'].values, params['ahead_days'].values))
    tic = time()
    
    train_val_time = int(params['train_val_time'])
    base_days = int(params['base_days'])
    step_days = int(params['step_days'])
    ahead_days = int(params['ahead_days'])
    
    print('Generating: base{}_ahead{}'.format(base_days, ahead_days))
    pid = 'base{}_ahead{}'.format(base_days, ahead_days)
    
    # Getting the data
    data_df = pd.read_pickle('../../data/data_train_val_df.pkl')
    today = data_df.index[-1]  # Real date
    print(pid + ') data_df loaded')
    # Drop symbols with many missing points
    data_df = pp.drop_irrelevant_symbols(data_df, params['GOOD_DATA_RATIO'])
    print(pid + ') Irrelevant symbols dropped.')
    
    # Generate the intervals for the predictor
    x, y = fe.generate_train_intervals(data_df, 
                                       train_val_time, 
                                       base_days, 
                                       step_days,
                                       ahead_days, 
                                       today, 
                                       fe.feature_close_one_to_one)    
    print(pid + ') Intervals generated')
    
    # Drop "bad" samples and fill missing data
    x_y_df = pd.concat([x, y], axis=1)
    x_y_df = pp.drop_irrelevant_samples(x_y_df, params['SAMPLES_GOOD_DATA_RATIO'])
    x = x_y_df.iloc[:, :-1]
    y = x_y_df.iloc[:, -1]
    x = pp.fill_missing(x)
    print(pid + ') Irrelevant samples dropped and missing data filled.')
    
    # Pickle that
    x.to_pickle('../../data/x_{}.pkl'.format(pid))
    y.to_pickle('../../data/y_{}.pkl'.format(pid))
    
    toc = time()
    print('%s) %i intervals generated in: %i seconds.' % (pid, x.shape[0], (toc-tic)))
    
    return pid, x, y
for ind in range(datasets_params_list_df.shape[0]):
    pid, x, y = generate_one_set(datasets_params_list_df.iloc[ind,:])
datasets_params_list_df['x_filename'] = datasets_params_list_df.apply(lambda x: 
                                                                      'x_base{}_ahead{}.pkl'.format(int(x['base_days']), 
                                                                                                    int(x['ahead_days'])), axis=1)
datasets_params_list_df['y_filename'] = datasets_params_list_df.apply(lambda x: 
                                                                      'y_base{}_ahead{}.pkl'.format(int(x['base_days']), 
                                                                                                    int(x['ahead_days'])), axis=1)
datasets_params_list_df
datasets_params_list_df.to_pickle('../../data/datasets_params_list_df.pkl')
Explanation: Now, let's define the function to generate each dataset.
Note: The way to treat the missing data was carefully thought. Never the missing data is filled across samples. Some symbols are discarded before the intervals generation. Some intervals are later discarded. Only after that, and only within training sample intervals the missing data is filled. That is done first forward and then backwards, to preserve, as much as possible, the causality.
End of explanation
<END_TASK>