<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
Step2:  Network Traffic Forecasting with AutoTS
In telco, accurate forecast of KPIs (e.g. network traffic, utilizations, user experience, etc.) for communication networks ( 2G/3G/4G/5G/wired) can help predict network failures, allocate resource, or save energy. 
In this notebook, we demostrate a reference use case where we use the network traffic KPI(s) in the past to predict traffic KPI(s) in the future. We demostrate how to use AutoTS in project Chronos to do time series forecasting in an automated and distributed way.
For demonstration, we use the publicly available network traffic data repository maintained by the WIDE project and in particular, the network traffic traces aggregated every 2 hours (i.e. AverageRate in Mbps/Gbps and Total Bytes) in year 2018 and 2019 at the transit link of WIDE to the upstream ISP (dataset link). 
Helper functions
This section defines some helper functions to be used in the following procedures. You can refer to it later when they're used.
Step3:  Download raw dataset and load into dataframe
Now we download the dataset and load it into a pandas dataframe. Steps are as below. 
First, run the script get_data.sh to download the raw data. It will download the monthly aggregated traffic data in year 2018 and 2019 into data folder. The raw data contains aggregated network traffic (average MBPs and total bytes) as well as other metrics. 
Second, run extract_data.sh to extract relavant traffic KPI's from raw data, i.e. AvgRate for average use rate, and total for total bytes. The script will extract the KPI's with timestamps into data/data.csv.
Finally, use pandas to load data/data.csv into a dataframe as shown below
Step4:  Below are some example records of the data
Step5:  Data pre-processing
Now we need to do data cleaning and preprocessing on the raw data. Note that this part could vary for different dataset. 
For the network traffic data we're using, the processing contains 3 parts
Step6:  Here, we drop weeks with more than 3 consecutive missing values and fill other missing values remained.
Step7:  Plot the data to see how the KPI's look like
Step8:  Time series forecasting with AutoTS
AutoTS provides AutoML support for building end-to-end time series analysis pipelines (including automatic feature generation, model selection and hyperparameter tuning).
The general workflow using automated training contains below two steps. 
   1. create a AutoTSTrainer to train a TSPipeline, save it to file to use later or elsewhere if you wish.
   2. use TSPipeline to do prediction, evaluation, and incremental fitting as well. 
First, you need to initialize RayOnSpark before using auto training (i.e. AutoTSTrainer), and stop it after training finished. (Note RayOnSpark is not needed if you just use TSPipeline for inference, evaluation or incremental training.)
Step9:  Then we initialize a AutoTSTrainer.
* dt_col
Step10:  We can set some searching presets such as look_back which indicates the history time period we want to use for forecasting.
lookback can be an int which it is a fixed values, or can be a tuple to indicate the range for sampling.
Step11:  We need to split the data frame into train, validation and test data frame before training. You can use train_val_test_split as an easy way to finish it.
Step12:  Then we fit on train data and validation data. 
You can use recipe to specify searching method as well as other searching presets such as stop criteria .etc. The GridRandomRecipe here is a recipe that combines grid search with random search to find the best set of parameters. For more details, please refer to bigdl document here.
Step13:  We get a TSPipeline after training. Let's print the hyper paramters selected.
Note that past_seq_len is the lookback value that is automatically chosen
Step14:  Use it to do prediction, evaluation or incremental fitting.
Step15:  plot actual and prediction values for AvgRate KPI
Step16:  Calculate mean square error and the symetric mean absolute percentage error.
Step17:  You can save the pipeline to file and reload it to do incremental fitting or others.
Step18:  You can stop RayOnSpark after auto training.
Step19:  Next, we demonstrate how to do incremental fitting with your saved pipeline file.
First load saved pipeline file.
Step20:  Then do incremental fitting with TSPipeline.fit().We use validation data frame as additional data for demonstration. You can use your new data frame.
Step21:  predict and plot the result after incremental fitting.
Step22:  Calculate mean square error and the symetric mean absolute percentage error.
<ASSISTANT_TASK:>
Python Code:
def get_drop_dates_and_len(df, allow_missing_num=3):
    
    Find missing values and get records to drop
    
    missing_num = df.total.isnull().astype(int).groupby(df.total.notnull().astype(int).cumsum()).sum()
    drop_missing_num = missing_num[missing_num > allow_missing_num]
    drop_datetimes = df.iloc[drop_missing_num.index].index
    drop_len = drop_missing_num.values
    return drop_datetimes, drop_len
def rm_missing_weeks(start_dts, missing_lens, df):
    
    Drop weeks that contains more than 3 consecutive missing values.
    If consecutive missing values across weeks, we remove all the weeks.
     
    for start_time, l in zip(start_dts, missing_lens):
        start = start_time - pd.Timedelta(days=start_time.dayofweek)
        start = start.replace(hour=0, minute=0, second=0)
        start_week_end = start + pd.Timedelta(days=6)
        start_week_end = start_week_end.replace(hour=22, minute=0, second=0)
        end_time = start_time + l*pd.Timedelta(hours=2)
        if start_week_end < end_time:
            end = end_time + pd.Timedelta(days=6-end_time.dayofweek)
            end = end.replace(hour=22, minute=0, second=0)
        else:
            end = start_week_end
        df = df.drop(df[start:end].index)
    return df
# plot the predicted values and actual values (for the test data)
def plot_result(test_df, pred_df, dt_col="datetime", value_col="AvgRate", look_back=1):
    # target column of dataframe is "value"
    # past sequence length is 50
    pred_value = pred_df[value_col][:-1].values
    true_value = test_df[value_col].values[look_back:]
    fig, axs = plt.subplots(figsize=(12, 5))
    axs.plot(pred_df[dt_col][:-1], pred_value, color='red', label='predicted values')
    axs.plot(test_df[dt_col][look_back:], true_value, color='blue', label='actual values')
    axs.set_title('the predicted values and actual values (for the test data)')
    plt.xlabel(dt_col)
    plt.xticks(rotation=45)
    plt.ylabel(value_col)
    plt.legend(loc='upper left')
    plt.show()
Explanation: Network Traffic Forecasting with AutoTS
In telco, accurate forecast of KPIs (e.g. network traffic, utilizations, user experience, etc.) for communication networks ( 2G/3G/4G/5G/wired) can help predict network failures, allocate resource, or save energy. 
In this notebook, we demostrate a reference use case where we use the network traffic KPI(s) in the past to predict traffic KPI(s) in the future. We demostrate how to use AutoTS in project Chronos to do time series forecasting in an automated and distributed way.
For demonstration, we use the publicly available network traffic data repository maintained by the WIDE project and in particular, the network traffic traces aggregated every 2 hours (i.e. AverageRate in Mbps/Gbps and Total Bytes) in year 2018 and 2019 at the transit link of WIDE to the upstream ISP (dataset link). 
Helper functions
This section defines some helper functions to be used in the following procedures. You can refer to it later when they're used.
End of explanation
import os 
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
%matplotlib inline
raw_df = pd.read_csv("data/data.csv")
Explanation: Download raw dataset and load into dataframe
Now we download the dataset and load it into a pandas dataframe. Steps are as below. 
First, run the script get_data.sh to download the raw data. It will download the monthly aggregated traffic data in year 2018 and 2019 into data folder. The raw data contains aggregated network traffic (average MBPs and total bytes) as well as other metrics. 
Second, run extract_data.sh to extract relavant traffic KPI's from raw data, i.e. AvgRate for average use rate, and total for total bytes. The script will extract the KPI's with timestamps into data/data.csv.
Finally, use pandas to load data/data.csv into a dataframe as shown below
End of explanation
raw_df.head()
Explanation: Below are some example records of the data
End of explanation
df = pd.DataFrame(pd.to_datetime(raw_df.StartTime))
# we can find 'AvgRate' is of two scales: 'Mbps' and 'Gbps' 
raw_df.AvgRate.str[-4:].unique()
# Unify AvgRate value
df['AvgRate'] = raw_df.AvgRate.apply(lambda x: float(x[:-4]) if x.endswith("Mbps") else float(x[:-4]) * 1000)
df["total"] = raw_df["total"]
df.set_index("StartTime", inplace=True)
full_idx = pd.date_range(start=df.index.min(), end=df.index.max(), freq='2H')
df = df.reindex(full_idx)
print("no. of n/a values:")
print(df.isna().sum())
Explanation: Data pre-processing
Now we need to do data cleaning and preprocessing on the raw data. Note that this part could vary for different dataset. 
For the network traffic data we're using, the processing contains 3 parts:
1. Convert string datetime to TimeStamp
2. Unify the measurement scale for AvgRate value - some uses Mbps, some uses Gbps 
3. Handle missing data (fill or drop).
End of explanation
drop_dts, drop_len = get_drop_dates_and_len(df)
df = rm_missing_weeks(drop_dts, drop_len, df)
df.ffill(inplace=True)
# AutoTS requires input data frame with a datetime column
df.index.name = "datetime"
df = df.reset_index()
df.head()
df.describe()
Explanation: Here, we drop weeks with more than 3 consecutive missing values and fill other missing values remained.
End of explanation
ax = df.plot(y='AvgRate',figsize=(12,5), title="AvgRate of network traffic data")
ax = df.plot(y='total',figsize=(12,5), title="total bytes of network traffic data")
Explanation: Plot the data to see how the KPI's look like
End of explanation
# init RayOnSpark in local mode
from bigdl.dllib.nncontext import init_spark_on_local
from bigdl.orca.ray import OrcaRayContext
sc = init_spark_on_local(cores=4, spark_log_level="INFO")
ray_ctx = OrcaRayContext(sc=sc, object_store_memory="1g")
ray_ctx.init()
Explanation: Time series forecasting with AutoTS
AutoTS provides AutoML support for building end-to-end time series analysis pipelines (including automatic feature generation, model selection and hyperparameter tuning).
The general workflow using automated training contains below two steps. 
   1. create a AutoTSTrainer to train a TSPipeline, save it to file to use later or elsewhere if you wish.
   2. use TSPipeline to do prediction, evaluation, and incremental fitting as well. 
First, you need to initialize RayOnSpark before using auto training (i.e. AutoTSTrainer), and stop it after training finished. (Note RayOnSpark is not needed if you just use TSPipeline for inference, evaluation or incremental training.)
End of explanation
from bigdl.chronos.autots.deprecated.forecast import AutoTSTrainer
trainer = AutoTSTrainer(dt_col="datetime",
                        target_col="AvgRate",
                        horizon=1,
                        extra_features_col=None)
Explanation: Then we initialize a AutoTSTrainer.
* dt_col: the column specifying datetime.
* target_col: target column to predict. Here, we take AvgRate KPI as an example.
* horizon : num of steps to look forward. 
* extra_feature_col: a list of columns which are also included in input data frame as features except target column.
End of explanation
# look back in range from one week to 3 days to predict the next 2h.
look_back = (36, 84)
Explanation: We can set some searching presets such as look_back which indicates the history time period we want to use for forecasting.
lookback can be an int which it is a fixed values, or can be a tuple to indicate the range for sampling.
End of explanation
from bigdl.chronos.autots.deprecated.preprocessing.utils import train_val_test_split
train_df, val_df, test_df = train_val_test_split(df, 
                                                 val_ratio=0.1, 
                                                 test_ratio=0.1,
                                                 look_back=look_back[0])
Explanation: We need to split the data frame into train, validation and test data frame before training. You can use train_val_test_split as an easy way to finish it.
End of explanation
from bigdl.chronos.autots.deprecated.config.recipe import LSTMGridRandomRecipe
%%time
ts_pipeline = trainer.fit(train_df, val_df, 
                          recipe=LSTMGridRandomRecipe(
                              num_rand_samples=1,
                              epochs=1,
                              look_back=look_back, 
                              batch_size=[64]),
                          metric="mse")
Explanation: Then we fit on train data and validation data. 
You can use recipe to specify searching method as well as other searching presets such as stop criteria .etc. The GridRandomRecipe here is a recipe that combines grid search with random search to find the best set of parameters. For more details, please refer to bigdl document here.
End of explanation
ts_pipeline.internal.config
Explanation: We get a TSPipeline after training. Let's print the hyper paramters selected.
Note that past_seq_len is the lookback value that is automatically chosen
End of explanation
pred_df = ts_pipeline.predict(test_df)
Explanation: Use it to do prediction, evaluation or incremental fitting.
End of explanation
# plot the predicted values and actual values
plot_result(test_df, pred_df, dt_col="datetime", value_col="AvgRate", look_back=ts_pipeline.internal.config['past_seq_len'])
Explanation: plot actual and prediction values for AvgRate KPI
End of explanation
mse, smape = ts_pipeline.evaluate(test_df, metrics=["mse", "smape"])
print("Evaluate: the mean square error is", mse)
print("Evaluate: the smape value is", smape)
Explanation: Calculate mean square error and the symetric mean absolute percentage error.
End of explanation
# save pipeline file
my_ppl_file_path = ts_pipeline.save("/tmp/saved_pipeline/my.ppl")
Explanation: You can save the pipeline to file and reload it to do incremental fitting or others.
End of explanation
# stop
ray_ctx.stop()
sc.stop()
Explanation: You can stop RayOnSpark after auto training.
End of explanation
# load file
from bigdl.chronos.autots.deprecated.forecast import TSPipeline
loaded_ppl = TSPipeline.load(my_ppl_file_path)
Explanation: Next, we demonstrate how to do incremental fitting with your saved pipeline file.
First load saved pipeline file.
End of explanation
# we use validation data frame as additional data for demonstration. 
loaded_ppl.fit(val_df, epochs=2)
Explanation: Then do incremental fitting with TSPipeline.fit().We use validation data frame as additional data for demonstration. You can use your new data frame.
End of explanation
# predict results of test_df
new_pred_df = loaded_ppl.predict(test_df)
plot_result(test_df, new_pred_df, look_back=loaded_ppl.internal.config['past_seq_len'])
Explanation: predict and plot the result after incremental fitting.
End of explanation
# evaluate test_df
mse, smape = loaded_ppl.evaluate(test_df, metrics=["mse", "smape"])
print("Evaluate: the mean square error is", mse)
print("Evaluate: the smape value is", smape)
Explanation: Calculate mean square error and the symetric mean absolute percentage error.
End of explanation
<END_TASK>