<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Introduction
Purposes
This tutorial
Step1:  To continue, read what OnTheFlySelector class is.
Step2:  Finally, let us print the list of parameters that can be passed when a new instance of OnTheFlySelector class is being created.
Step3:  Application
Import Statements
Step4:  Data Extraction and Brief Exploration
The dataset that is used here is a set of synthetic time series that are drawn from a generative model trained on lots of real-world time series, so the problem under consideration is quite realistic.
First of all, download the dataset if it has not been downloaded before.
Step5:  How many time series are there?
Step6:  Each time series includes two months of observations.
Metrics
Now let us define some metrics, but before that a quick remark is here — so much attention is paid to this section, because it explains which properties one can expect from on-the-fly selection and which properties, conversely, one can not expect from it.
An interesting combination is to use both $R^2$ coefficient computed in a batch for all time series and $R^2$ coefficient computed for each time series separately and then averaged over all of them. The former metric reports how well levels of different time series are grasped, whereas the latter one reports how well individual dynamics and deviations from a corresponding mean are predicted.
In addition, MAPE (mean absolute percentage error) computed for each time series separately, censored from above at 100% level, and averaged over all time series, can be displayed too, because it shows how far predictions are from actual values in relative terms.
To see why these three metrics differ, look at the below example.
Step7:  Above metric is high, because two series from example_df have different levels and predictions are near the corresponding levels which means that variation across levels is reflected in predictions.
Step8:  Alas, this metric is negative, because variation around individual means is not reflected at all.
Step9:  Finally, value of the third metric is neither decent nor poor. Predictions are not too far from actual values — relative difference is about 24%.
The Launch Itself
Step10:  Since 1 is the maximal possible value of $R^2$, the above score is very good.
Step11:  Negative value indicates that true mean is better than the forecast.
<ASSISTANT_TASK:>
Python Code:
from utils_for_demo_of_on_the_fly_selector import Helper
Helper().why_adaptive_selection()
Explanation: Introduction
Purposes
This tutorial:
* provides a theoretical description of adaptive selection;
* demonstrates how to use OnTheFlySelector for a problem of forecasting large amount of time series;
* suggests a set of metrics such that properties of on-the-fly selection are revealed by them.
Background
In order to avoid copying and pasting sentences from docstrings, let us extract and format all necessary information. This is done with a class that hides too specialized code under easy-to-read names.
End of explanation
Helper().what_is_on_the_fly_selector()
Explanation: To continue, read what OnTheFlySelector class is.
End of explanation
Helper().which_parameters_does_on_the_fly_selector_have()
Explanation: Finally, let us print the list of parameters that can be passed when a new instance of OnTheFlySelector class is being created.
End of explanation
import os
import datetime
import pandas as pd
from forecastonishing.selection.on_the_fly_selector import OnTheFlySelector
from forecastonishing.selection.paralleling import (
    fit_selector_in_parallel,
    predict_with_selector_in_parallel
)
from forecastonishing.miscellaneous.metrics import (
    overall_r_squared,
    averaged_r_squared,
    averaged_censored_mape
)
Explanation: Application
Import Statements
End of explanation
path_to_dataset = 'time_series_dataset.csv'
if os.path.isfile(path_to_dataset):
    df = pd.read_csv(path_to_dataset, parse_dates=[2])
else:
    df = pd.read_csv(
        "https://docs.google.com/spreadsheets/" +
        "d/1TF0bAf9wOpIXIvIsazMCLEoHQ1y6dTkYYdYRRleC5lM/export?format=csv",
        parse_dates=[2]
    )
    df.to_csv(path_to_dataset, index=False)
df.head()
Explanation: Data Extraction and Brief Exploration
The dataset that is used here is a set of synthetic time series that are drawn from a generative model trained on lots of real-world time series, so the problem under consideration is quite realistic.
First of all, download the dataset if it has not been downloaded before.
End of explanation
n_time_series = len(df.groupby(['unit', 'item']))
n_time_series
len(df.index) / n_time_series
Explanation: How many time series are there?
End of explanation
example_df = pd.DataFrame(
    [[1, 2, 3],
     [1, 4, 5],
     [2, 10, 9],
     [2, 9, 10]],
    columns=['key', 'actual_value', 'prediction']
)
overall_r_squared(example_df)
Explanation: Each time series includes two months of observations.
Metrics
Now let us define some metrics, but before that a quick remark is here — so much attention is paid to this section, because it explains which properties one can expect from on-the-fly selection and which properties, conversely, one can not expect from it.
An interesting combination is to use both $R^2$ coefficient computed in a batch for all time series and $R^2$ coefficient computed for each time series separately and then averaged over all of them. The former metric reports how well levels of different time series are grasped, whereas the latter one reports how well individual dynamics and deviations from a corresponding mean are predicted.
In addition, MAPE (mean absolute percentage error) computed for each time series separately, censored from above at 100% level, and averaged over all time series, can be displayed too, because it shows how far predictions are from actual values in relative terms.
To see why these three metrics differ, look at the below example.
End of explanation
averaged_r_squared(example_df, ['key'])
Explanation: Above metric is high, because two series from example_df have different levels and predictions are near the corresponding levels which means that variation across levels is reflected in predictions.
End of explanation
averaged_censored_mape(example_df, ['key'])
Explanation: Alas, this metric is negative, because variation around individual means is not reflected at all.
End of explanation
horizon = 3
n_evaluational_rounds = 10
# This is not a parameter of `OnTheFlySelector` or any of its methods.
# It is introduced, because, by default, `OnTheFlySelector` does not
# use older lags, so it allows filtering redundant observations.
max_lag_to_use = 10
train_test_frontier = df['date'].max() - datetime.timedelta(days=horizon-1)
n_training_days = horizon + n_evaluational_rounds + max_lag_to_use + 1
train_df = df[
    (df['date'] < train_test_frontier) &
    (df['date'] >= train_test_frontier - datetime.timedelta(days=n_training_days))
]
test_df = df[df['date'] >= train_test_frontier]
selector = OnTheFlySelector(
    horizon=horizon,
    n_evaluational_rounds=n_evaluational_rounds,
    verbose=1
)
%%time
selector = fit_selector_in_parallel(
    selector,
    train_df,
    name_of_target='value',
    series_keys=['unit', 'item'],
    n_processes=4
)
%%time
predictions_df = predict_with_selector_in_parallel(
    selector,
    train_df,
    n_processes=4
)
# Sorting is necessary after parallel execution.
evaluation_df = predictions_df.reset_index().sort_values(by=['unit', 'item', 'index'])
evaluation_df['actual_value'] = test_df.sort_values(by=['unit', 'item', 'date'])['value'].values
overall_r_squared(evaluation_df)
Explanation: Finally, value of the third metric is neither decent nor poor. Predictions are not too far from actual values — relative difference is about 24%.
The Launch Itself
End of explanation
averaged_r_squared(evaluation_df, ['unit', 'item'])
Explanation: Since 1 is the maximal possible value of $R^2$, the above score is very good.
End of explanation
averaged_censored_mape(evaluation_df, ['unit', 'item'])
Explanation: Negative value indicates that true mean is better than the forecast.
End of explanation
<END_TASK>