<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
Step1:  Network Traffic Forecasting (using time series data)
In telco, accurate forecast of KPIs (e.g. network traffic, utilizations, user experience, etc.) for communication networks ( 2G/3G/4G/5G/wired) can help predict network failures, allocate resource, or save energy. 
In this notebook, we demonstrate a reference use case where we use the network traffic KPI(s) in the past to predict traffic KPI(s) in the future. We demostrate how to do univariate forecasting (predict only 1 series), and multivariate forecasting (predicts more than 1 series at the same time) using Project Chronos.
For demonstration, we use the publicly available network traffic data repository maintained by the WIDE project and in particular, the network traffic traces aggregated every 2 hours (i.e. AverageRate in Mbps/Gbps and Total Bytes) in year 2018 and 2019 at the transit link of WIDE to the upstream ISP (dataset link). 
Helper functions
This section defines some helper functions to be used in the following procedures. You can refer to it later when they're used.
Step2:  Download raw dataset and load into dataframe
Now we download the dataset and load it into a pandas dataframe. Steps are as below. 
First, run the script get_data.sh to download the raw data. It will download the monthly aggregated traffic data in year 2018 and 2019 into data folder. The raw data contains aggregated network traffic (average MBPs and total bytes) as well as other metrics. 
Second, run extract_data.sh to extract relavant traffic KPI's from raw data, i.e. AvgRate for average use rate, and total for total bytes. The script will extract the KPI's with timestamps into data/data.csv.
Finally, use pandas to load data/data.csv into a dataframe as shown below
Step3:  Below are some example records of the data
Step4:  Data pre-processing
Now we need to do data cleaning and preprocessing on the raw data. Note that this part could vary for different dataset. 
For the network traffic data we're using, the processing contains 2 parts
Step5:  Plot the data to see how the KPI's look like
Step6:  Feature Engineering & Data Preperation
For feature engineering, we use year, month, week, day of week and hour as features in addition to the target KPI values.
For data preperation, we impute the data to handle missing data and scale the data.
We generate a built-in TSDataset to complete the whole processing.
Step7:  Time series forecasting
Univariate forecasting
For univariate forecasting, we forecast AvgRate only. We need to roll the data on corresponding target column with tsdataset.
Step8:  For univariate forcasting, we use LSTMForecaster for forecasting.
Step9:  First we initiate a LSTMForecaster. 
* feature_dim should match the training data input feature, so we just use the last dimension of train data shape. 
* target_dim equals the variate num we want to predict. We set target_dim=1 for univariate forecasting.
Step10:  Then we use fit to train the model. Wait sometime for it to finish.
Step11:  After training is finished. You can use the forecaster to do prediction and evaluation.
Step12:  Since we have used standard scaler to scale the input data (including the target values), we need to inverse the scaling on the predicted values too.
Step13:  calculate the symetric mean absolute percentage error.
Step14:  multivariate forecasting
For multivariate forecasting, we forecast AvgRate and total at the same time. We need to roll the data on corresponding target column with tsdataset.
Step15:  For multivariate forecasting, we use MTNetForecaster for forecasting.
Step16:  First, we initialize a mtnet_forecaster according to input data shape. The lookback length is equal to (long_series_num+1)*series_length Details refer to chronos docs.
Step17:  Now we train the model and wait till it finished.
Step18:  Use the model for prediction and inverse the scaling of the prediction results
Step19:  plot actual and prediction values for AvgRate KPI
Step20:  plot actual and prediction values for total bytes KPI
<ASSISTANT_TASK:>
Python Code:
def plot_predict_actual_values(date, y_pred, y_test, ylabel):
    
    plot the predicted values and actual values (for the test data)
    
    fig, axs = plt.subplots(figsize=(16,6))
    axs.plot(date, y_pred, color='red', label='predicted values')
    axs.plot(date, y_test, color='blue', label='actual values')
    axs.set_title('the predicted values and actual values (for the test data)')
    plt.xlabel('test datetime')
    plt.ylabel(ylabel)
    plt.legend(loc='upper left')
    plt.show()
Explanation: Network Traffic Forecasting (using time series data)
In telco, accurate forecast of KPIs (e.g. network traffic, utilizations, user experience, etc.) for communication networks ( 2G/3G/4G/5G/wired) can help predict network failures, allocate resource, or save energy. 
In this notebook, we demonstrate a reference use case where we use the network traffic KPI(s) in the past to predict traffic KPI(s) in the future. We demostrate how to do univariate forecasting (predict only 1 series), and multivariate forecasting (predicts more than 1 series at the same time) using Project Chronos.
For demonstration, we use the publicly available network traffic data repository maintained by the WIDE project and in particular, the network traffic traces aggregated every 2 hours (i.e. AverageRate in Mbps/Gbps and Total Bytes) in year 2018 and 2019 at the transit link of WIDE to the upstream ISP (dataset link). 
Helper functions
This section defines some helper functions to be used in the following procedures. You can refer to it later when they're used.
End of explanation
import os 
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
%matplotlib inline
raw_df = pd.read_csv("data/data.csv")
Explanation: Download raw dataset and load into dataframe
Now we download the dataset and load it into a pandas dataframe. Steps are as below. 
First, run the script get_data.sh to download the raw data. It will download the monthly aggregated traffic data in year 2018 and 2019 into data folder. The raw data contains aggregated network traffic (average MBPs and total bytes) as well as other metrics. 
Second, run extract_data.sh to extract relavant traffic KPI's from raw data, i.e. AvgRate for average use rate, and total for total bytes. The script will extract the KPI's with timestamps into data/data.csv.
Finally, use pandas to load data/data.csv into a dataframe as shown below
End of explanation
raw_df.head()
Explanation: Below are some example records of the data
End of explanation
df = pd.DataFrame(pd.to_datetime(raw_df.StartTime))
# we can find 'AvgRate' is of two scales: 'Mbps' and 'Gbps'
raw_df.AvgRate.str[-4:].unique() 
# Unify AvgRate value
df['AvgRate'] = raw_df.AvgRate.apply(lambda x:float(x[:-4]) if x.endswith("Mbps") else float(x[:-4])*1000)
df["total"] = raw_df["total"]
df.head()
Explanation: Data pre-processing
Now we need to do data cleaning and preprocessing on the raw data. Note that this part could vary for different dataset. 
For the network traffic data we're using, the processing contains 2 parts:
1. Convert string datetime to TimeStamp
2. Unify the measurement scale for AvgRate value - some uses Mbps, some uses Gbps
End of explanation
ax = df.plot(y='AvgRate',figsize=(16,6), title="AvgRate of network traffic data")
ax = df.plot(y='total',figsize=(16,6), title="total bytes of network traffic data")
Explanation: Plot the data to see how the KPI's look like
End of explanation
from bigdl.chronos.data import TSDataset
from sklearn.preprocessing import StandardScaler
# we look back one week data which is of the frequency of 2h.
look_back = 84
# specify the number of steps to be predicted，one day is selected by default.
horizon = 1
tsdata_train, _, tsdata_test = TSDataset.from_pandas(df, dt_col="StartTime", target_col=["AvgRate","total"], with_split=True, test_ratio=0.1)
standard_scaler = StandardScaler()
for tsdata in [tsdata_train, tsdata_test]:
    tsdata.gen_dt_feature()\
          .impute(mode="last")\
          .scale(standard_scaler, fit=(tsdata is tsdata_train))
Explanation: Feature Engineering & Data Preperation
For feature engineering, we use year, month, week, day of week and hour as features in addition to the target KPI values.
For data preperation, we impute the data to handle missing data and scale the data.
We generate a built-in TSDataset to complete the whole processing.
End of explanation
for tsdata in [tsdata_train, tsdata_test]:
    tsdata.roll(lookback=look_back, horizon=horizon, target_col="AvgRate")
x_train, y_train = tsdata_train.to_numpy()
x_test, y_test = tsdata_test.to_numpy()
x_train.shape, y_train.shape, x_test.shape, y_test.shape    
Explanation: Time series forecasting
Univariate forecasting
For univariate forecasting, we forecast AvgRate only. We need to roll the data on corresponding target column with tsdataset.
End of explanation
from bigdl.chronos.forecaster.lstm_forecaster import LSTMForecaster
Explanation: For univariate forcasting, we use LSTMForecaster for forecasting.
End of explanation
# build model
forecaster = LSTMForecaster(past_seq_len=x_train.shape[1],
                            input_feature_num=x_train.shape[-1],
                            output_feature_num=y_train.shape[-1],
                            hidden_dim=16,
                            layer_num=2,
                            lr=0.001)
Explanation: First we initiate a LSTMForecaster. 
* feature_dim should match the training data input feature, so we just use the last dimension of train data shape. 
* target_dim equals the variate num we want to predict. We set target_dim=1 for univariate forecasting.
End of explanation
%%time
forecaster.fit(data=(x_train, y_train), batch_size=1024, epochs=50)
Explanation: Then we use fit to train the model. Wait sometime for it to finish.
End of explanation
# make prediction
y_pred = forecaster.predict(x_test)
Explanation: After training is finished. You can use the forecaster to do prediction and evaluation.
End of explanation
y_pred_unscale = tsdata_test.unscale_numpy(y_pred)
y_test_unscale = tsdata_test.unscale_numpy(y_test)
Explanation: Since we have used standard scaler to scale the input data (including the target values), we need to inverse the scaling on the predicted values too.
End of explanation
from bigdl.orca.automl.metrics import Evaluator
# evaluate with sMAPE
print("sMAPE is", Evaluator.evaluate("smape", y_test_unscale, y_pred_unscale))
# evaluate with mean_squared_error
print("mean_squared error is", Evaluator.evaluate("mse", y_test_unscale, y_pred_unscale))
Explanation: calculate the symetric mean absolute percentage error.
End of explanation
for tsdata in [tsdata_train, tsdata_test]:
    tsdata.roll(lookback=look_back, horizon=horizon, target_col=["AvgRate","total"])
    
x_train_m, y_train_m = tsdata_train.to_numpy()
x_test_m, y_test_m = tsdata_test.to_numpy()
y_train_m, y_test_m = y_train_m[:, 0, :], y_test_m[:, 0, :]
x_train_m.shape, y_train_m.shape, x_test_m.shape, y_test_m.shape
Explanation: multivariate forecasting
For multivariate forecasting, we forecast AvgRate and total at the same time. We need to roll the data on corresponding target column with tsdataset.
End of explanation
from bigdl.chronos.forecaster.tf.mtnet_forecaster import MTNetForecaster
Explanation: For multivariate forecasting, we use MTNetForecaster for forecasting.
End of explanation
mtnet_forecaster = MTNetForecaster(target_dim=y_train_m.shape[-1],
                                   feature_dim=x_train_m.shape[-1],
                                   long_series_num=6,
                                   series_length=12,
                                   ar_window_size=6,
                                   cnn_height=4)
Explanation: First, we initialize a mtnet_forecaster according to input data shape. The lookback length is equal to (long_series_num+1)*series_length Details refer to chronos docs.
End of explanation
%%time
hist = mtnet_forecaster.fit(data=(x_train_m, y_train_m), batch_size=1024, epochs=20)
Explanation: Now we train the model and wait till it finished.
End of explanation
y_pred_m = mtnet_forecaster.predict(x_test_m)
y_pred_m_unscale = tsdata_test.unscale_numpy(np.expand_dims(y_pred_m, axis=1))[:, 0, :]
y_test_m_unscale = tsdata_test.unscale_numpy(np.expand_dims(y_test_m, axis=1))[:, 0, :]
from bigdl.chronos.metric.forecast_metrics import Evaluator
res = Evaluator.evaluate(["smape", "mse"],
                         y_true=y_test_m_unscale,
                         y_pred=y_pred_m_unscale)
# evaluate with sMAPE
print("sMAPE is", res[0])
# evaluate with mean_squared_error
print("mean_squared error is", res[1])
Explanation: Use the model for prediction and inverse the scaling of the prediction results
End of explanation
multi_target_value = ["AvgRate","total"]
test_date=df[-y_test_m_unscale.shape[0]:].index
plot_predict_actual_values(test_date, y_pred_m_unscale[:,0], y_test_m_unscale[:,0], ylabel=multi_target_value[0])
Explanation: plot actual and prediction values for AvgRate KPI
End of explanation
plot_predict_actual_values(test_date, y_pred_m_unscale[:,1], y_test_m_unscale[:,1], ylabel=multi_target_value[1])
Explanation: plot actual and prediction values for total bytes KPI
End of explanation
<END_TASK>