<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Introduction
Run this cell to set everything up!
Step1:  Examine the following seasonal plot
Step2:  And also the periodogram
Step3:  1) Determine seasonality
What kind of seasonality do you see evidence of? Once you've thought about it, run the next cell for some discussion.
Step4:  2) Create seasonal features
Use DeterministicProcess and CalendarFourier to create
Step5:  Now run this cell to fit the seasonal model.
Step6:  Removing from a series its trend or seasons is called detrending or deseasonalizing the series.
Look at the periodogram of the deseasonalized series.
Step7:  3) Check for remaining seasonality
Based on these periodograms, how effectively does it appear your model captured the seasonality in Average Sales? Does the periodogram agree with the time plot of the deseasonalized series?
Step8:  The Store Sales dataset includes a table of Ecuadorian holidays.
Step9:  From a plot of the deseasonalized Average Sales, it appears these holidays could have some predictive power.
Step10:  4) Create holiday features
What kind of features could you create to help your model make use of this information? Code your answer in the next cell. (Scikit-learn and Pandas both have utilities that should make this easy. See the hint if you'd like more details.)
Step11:  Use this cell to fit the seasonal model with holiday features added. Do the fitted values seem to have improved?
Step12:  (Optional) Submit to Store Sales competition
This part of the exercise will walk you through your first submission to this course's companion competition
Step13:  You can use this cell to see some of its predictions.
Step14:  Finally, this cell loads the test data, creates a feature set for the forecast period, and then creates the submission file submission.csv.
<ASSISTANT_TASK:>
Python Code:
# Setup feedback system
from learntools.core import binder
binder.bind(globals())
from learntools.time_series.ex3 import *
# Setup notebook
from pathlib import Path
from learntools.time_series.style import *  # plot style settings
from learntools.time_series.utils import plot_periodogram, seasonal_plot
import pandas as pd
import matplotlib.pyplot as plt
import numpy as np
import seaborn as sns
from sklearn.linear_model import LinearRegression
from statsmodels.tsa.deterministic import CalendarFourier, DeterministicProcess
comp_dir = Path('../input/store-sales-time-series-forecasting')
holidays_events = pd.read_csv(
    comp_dir / "holidays_events.csv",
    dtype={
        'type': 'category',
        'locale': 'category',
        'locale_name': 'category',
        'description': 'category',
        'transferred': 'bool',
    },
    parse_dates=['date'],
    infer_datetime_format=True,
)
holidays_events = holidays_events.set_index('date').to_period('D')
store_sales = pd.read_csv(
    comp_dir / 'train.csv',
    usecols=['store_nbr', 'family', 'date', 'sales'],
    dtype={
        'store_nbr': 'category',
        'family': 'category',
        'sales': 'float32',
    },
    parse_dates=['date'],
    infer_datetime_format=True,
)
store_sales['date'] = store_sales.date.dt.to_period('D')
store_sales = store_sales.set_index(['store_nbr', 'family', 'date']).sort_index()
average_sales = (
    store_sales
    .groupby('date').mean()
    .squeeze()
    .loc['2017']
)
Explanation: Introduction
Run this cell to set everything up!
End of explanation
X = average_sales.to_frame()
X["week"] = X.index.week
X["day"] = X.index.dayofweek
seasonal_plot(X, y='sales', period='week', freq='day');
Explanation: Examine the following seasonal plot:
End of explanation
plot_periodogram(average_sales);
Explanation: And also the periodogram:
End of explanation
# View the solution (Run this cell to receive credit!)
q_1.check()
Explanation: 1) Determine seasonality
What kind of seasonality do you see evidence of? Once you've thought about it, run the next cell for some discussion.
End of explanation
y = average_sales.copy()
# YOUR CODE HERE
fourier = ____
dp = DeterministicProcess(
    index=y.index,
    constant=True,
    order=1,
    # YOUR CODE HERE
    # ____
    drop=True,
)
X = ____
# Check your answer
q_2.check()
# Lines below will give you a hint or solution code
#_COMMENT_IF(PROD)_
q_2.hint()
#_COMMENT_IF(PROD)_
q_2.solution()
#%%RM_IF(PROD)%%
y = average_sales.copy()
fourier = CalendarFourier(freq='M', order=12)
dp = DeterministicProcess(
    index=y.index,
    constant=True,
    order=1,
    seasonal=True,
    additional_terms=[fourier],
    drop=True,
)
X = dp.in_sample()
q_2.assert_check_failed()
#%%RM_IF(PROD)%%
y = average_sales.copy()
fourier = CalendarFourier(freq='A', order=4)
dp = DeterministicProcess(
    index=y.index,
    constant=True,
    order=1,
    seasonal=True,
    additional_terms=[fourier],
    drop=True,
)
X = dp.in_sample()
q_2.assert_check_failed()
#%%RM_IF(PROD)%%
y = average_sales.copy()
fourier = CalendarFourier(freq='M', order=4)
dp = DeterministicProcess(
    index=y.index[1:],
    constant=True,
    order=1,
    seasonal=True,
    additional_terms=[fourier],
    drop=True,
)
X = dp.in_sample()
q_2.assert_check_failed()
#%%RM_IF(PROD)%%
y = average_sales.copy()
fourier = CalendarFourier(freq='M', order=4)
dp = DeterministicProcess(
    index=y.index[1:],
    constant=True,
    order=1,
    seasonal=False,
    additional_terms=[fourier],
    drop=True,
)
X = dp.in_sample()
q_2.assert_check_failed()
#%%RM_IF(PROD)%%
y = average_sales.copy()
fourier = CalendarFourier(freq='M', order=4)
dp = DeterministicProcess(
    index=y.index,
    constant=True,
    order=1,
    seasonal=True,
#    additional_terms=[fourier],
    drop=True,
)
X = dp.in_sample()
q_2.assert_check_failed()
#%%RM_IF(PROD)%%
y = average_sales.copy()
fourier = CalendarFourier(freq='M', order=4)
dp = DeterministicProcess(
    index=y.index,
    constant=True,
    order=1,
    seasonal=True,
    additional_terms=[fourier],
    drop=True,
)
X = dp.in_sample()
q_2.assert_check_passed()
Explanation: 2) Create seasonal features
Use DeterministicProcess and CalendarFourier to create:
- indicators for weekly seasons and
- Fourier features of order 4 for monthly seasons.
End of explanation
model = LinearRegression().fit(X, y)
y_pred = pd.Series(
    model.predict(X),
    index=X.index,
    name='Fitted',
)
y_pred = pd.Series(model.predict(X), index=X.index)
ax = y.plot(**plot_params, alpha=0.5, title="Average Sales", ylabel="items sold")
ax = y_pred.plot(ax=ax, label="Seasonal")
ax.legend();
Explanation: Now run this cell to fit the seasonal model.
End of explanation
y_deseason = y - y_pred
fig, (ax1, ax2) = plt.subplots(2, 1, sharex=True, sharey=True, figsize=(10, 7))
ax1 = plot_periodogram(y, ax=ax1)
ax1.set_title("Product Sales Frequency Components")
ax2 = plot_periodogram(y_deseason, ax=ax2);
ax2.set_title("Deseasonalized");
Explanation: Removing from a series its trend or seasons is called detrending or deseasonalizing the series.
Look at the periodogram of the deseasonalized series.
End of explanation
# View the solution (Run this cell to receive credit!)
q_3.check()
Explanation: 3) Check for remaining seasonality
Based on these periodograms, how effectively does it appear your model captured the seasonality in Average Sales? Does the periodogram agree with the time plot of the deseasonalized series?
End of explanation
# National and regional holidays in the training set
holidays = (
    holidays_events
    .query("locale in ['National', 'Regional']")
    .loc['2017':'2017-08-15', ['description']]
    .assign(description=lambda x: x.description.cat.remove_unused_categories())
)
display(holidays)
Explanation: The Store Sales dataset includes a table of Ecuadorian holidays.
End of explanation
ax = y_deseason.plot(**plot_params)
plt.plot_date(holidays.index, y_deseason[holidays.index], color='C3')
ax.set_title('National and Regional Holidays');
Explanation: From a plot of the deseasonalized Average Sales, it appears these holidays could have some predictive power.
End of explanation
# YOUR CODE HERE
#_UNCOMMENT_IF(PROD)_
#X_holidays = ____
#_UNCOMMENT_IF(PROD)_
#X2 = X.join(X_holidays, on='date').fillna(0.0)
# Check your answer
q_4.check()
# Lines below will give you a hint or solution code
#_COMMENT_IF(PROD)_
q_4.hint()
#_COMMENT_IF(PROD)_
q_4.hint(2)
#_COMMENT_IF(PROD)_
q_4.solution()
#%%RM_IF(PROD)%%
# Scikit-learn solution
from sklearn.preprocessing import OneHotEncoder
ohe = OneHotEncoder(sparse=False)
X_holidays = pd.DataFrame(
    ohe.fit_transform(holidays),
    index=holidays.index,
    columns=holidays.description.unique(),
)
X2 = X.join(X_holidays, on='date').fillna(0.0)
q_4.assert_check_passed()
#%%RM_IF(PROD)%%
# Pandas solution
X_holidays = pd.get_dummies(holidays)
X2 = X.join(X_holidays, on='date').fillna(0.0)
q_4.assert_check_passed()
Explanation: 4) Create holiday features
What kind of features could you create to help your model make use of this information? Code your answer in the next cell. (Scikit-learn and Pandas both have utilities that should make this easy. See the hint if you'd like more details.)
End of explanation
model = LinearRegression().fit(X2, y)
y_pred = pd.Series(
    model.predict(X2),
    index=X2.index,
    name='Fitted',
)
y_pred = pd.Series(model.predict(X2), index=X2.index)
ax = y.plot(**plot_params, alpha=0.5, title="Average Sales", ylabel="items sold")
ax = y_pred.plot(ax=ax, label="Seasonal")
ax.legend();
Explanation: Use this cell to fit the seasonal model with holiday features added. Do the fitted values seem to have improved?
End of explanation
y = store_sales.unstack(['store_nbr', 'family']).loc["2017"]
# Create training data
fourier = CalendarFourier(freq='M', order=4)
dp = DeterministicProcess(
    index=y.index,
    constant=True,
    order=1,
    seasonal=True,
    additional_terms=[fourier],
    drop=True,
)
X = dp.in_sample()
X['NewYear'] = (X.index.dayofyear == 1)
model = LinearRegression(fit_intercept=False)
model.fit(X, y)
y_pred = pd.DataFrame(model.predict(X), index=X.index, columns=y.columns)
Explanation: (Optional) Submit to Store Sales competition
This part of the exercise will walk you through your first submission to this course's companion competition: Store Sales - Time Series Forecasting. Submitting to the competition isn't required to complete the course, but it's a great way to try out your new skills.
The next cell creates a seasonal model of the kind you've learned about in this lesson for the full Store Sales dataset with all 1800 time series.
End of explanation
STORE_NBR = '1'  # 1 - 54
FAMILY = 'PRODUCE'
# Uncomment to see a list of product families
# display(store_sales.index.get_level_values('family').unique())
ax = y.loc(axis=1)['sales', STORE_NBR, FAMILY].plot(**plot_params)
ax = y_pred.loc(axis=1)['sales', STORE_NBR, FAMILY].plot(ax=ax)
ax.set_title(f'{FAMILY} Sales at Store {STORE_NBR}');
Explanation: You can use this cell to see some of its predictions.
End of explanation
df_test = pd.read_csv(
    comp_dir / 'test.csv',
    dtype={
        'store_nbr': 'category',
        'family': 'category',
        'onpromotion': 'uint32',
    },
    parse_dates=['date'],
    infer_datetime_format=True,
)
df_test['date'] = df_test.date.dt.to_period('D')
df_test = df_test.set_index(['store_nbr', 'family', 'date']).sort_index()
# Create features for test set
X_test = dp.out_of_sample(steps=16)
X_test.index.name = 'date'
X_test['NewYear'] = (X_test.index.dayofyear == 1)
y_submit = pd.DataFrame(model.predict(X_test), index=X_test.index, columns=y.columns)
y_submit = y_submit.stack(['store_nbr', 'family'])
y_submit = y_submit.join(df_test.id).reindex(columns=['id', 'sales'])
y_submit.to_csv('submission.csv', index=False)
Explanation: Finally, this cell loads the test data, creates a feature set for the forecast period, and then creates the submission file submission.csv.
End of explanation
<END_TASK>