<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Manipulating Time Series
Notebook to follow along with final chapter of the DataCamp Manipulating Time Series course.
Step1:  Import the data
Step2:  Problem
Step3:  Build a Market Cap Weighted Index
Step4:  Evaluate Index Performance
Step5:  Index Correlation and Exporting to Excel
Step6:  Time Series Analysis
Second course in Skill Track - look at AR, MA, ACF etc
<ASSISTANT_TASK:>
Python Code:
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import os
import yfinance as yf
%matplotlib inline
def ddir(name=None):
    data_dir = 'dc_manipulating_time_series/stock_data/'
    if name is None:
        print(os.listdir(data_dir))
    else:
        return os.path.join(data_dir, name)
ddir()
Explanation: Manipulating Time Series
Notebook to follow along with final chapter of the DataCamp Manipulating Time Series course.
End of explanation
nyse = pd.read_excel(ddir('listings.xlsx'), sheet_name='nyse', na_values='n/a')
nyse.info()
nyse.head()
nyse.loc[nyse['Stock Symbol'].isin(['CPRT', 'ILMN']), :]
Explanation: Import the data
End of explanation
# NASDAQ listings from https://www.nasdaq.com/market-activity/stocks/screener 
# Looks like the listings data for the exercise is from Nasdaq rather than NYSE
nasdaq = pd.read_csv(ddir('nasdaq_screener_1646066842549.csv')).rename(
    columns={'Symbol': 'Stock Symbol', 'Market Cap': 'Market Capitalization', 'Name': 'Company Name'}
)
nasdaq['Last Sale'] = nasdaq['Last Sale'].replace({'\$': '', ',': ''}, regex=True).astype(float)
nasdaq.head()
nasdaq.set_index('Stock Symbol', inplace=True) # Make Stock Symbol the index
nasdaq.dropna(subset=['Sector'], inplace=True) # Remove stocks without sector info
nasdaq['Market Capitalization'] /= 1e6 # Scale to million $ 
# Exercise only considers stocks that IPOd before 2019, need to filter
# by this as well otherwise reading stock time series data fails 
# due to missing tickers.  Also fiddle with sector label to get things matching.
nasdaq = nasdaq.loc[(nasdaq['IPO Year'] < 2016), :]
nasdaq.loc['GS', 'Sector'] = 'Bank'
nasdaq.loc['ILMN', 'Sector'] = 'Biotech'
nasdaq.loc['CPRT', 'Sector'] = 'Finance'
nasdaq.info()
# nasdaq.loc[nasdaq.index.isin(data.columns), :]
allowed_sectors = ['Technology', 'Health Care', 'Consumer Services', 'Miscellaneous',
 'Consumer Non-Durables', 'Bank', 'Health Care', 'Biotech', 'Energy', 'Basic Industries',
 'Public Utilities', 'Transportation']
disallowed_stocks = ['LIN', 'TSLA', 'GNRC', 'MPLX', 'HDB', 'ABBV', 'KMI', 'FANG', 'RSG',
                    'CLR', 'AWK', 'WES', 'PSXP']
components = nasdaq.loc[
    nasdaq.Sector.isin(allowed_sectors) & ~nasdaq.index.isin(disallowed_stocks), :
].groupby('Sector')['Market Capitalization'].nlargest(1)
components.sort_values(ascending=False)
components.info()
# Index is MultiIndex so use get_level_values to extract tickers
tickers = components.index.get_level_values('Stock Symbol')
tickers
columns = ['Company Name', 'Market Capitalization', 'Last Sale']
component_info = nasdaq.loc[tickers, columns].sort_values('Market Capitalization', ascending=False)
pd.options.display.float_format = '{:,.2f}'.format
component_info
# Read the price time series data 
data = pd.read_csv(
    ddir('stock_data.csv'), parse_dates=['Date'], index_col='Date'
).loc[:, tickers.tolist()].dropna()
data.info()
# Look at price returns over period
price_return = data.iloc[-1].div(data.iloc[0]).sub(1).mul(100).sort_values(ascending=False)
price_return
price_return.sort_values().plot(title='Stock Price Returns', kind='barh')
plt.show()
Explanation: Problem: it looks like the listing data Excel file from the Datacamp data download for this course contains different stocks to the ones we want.  Some investigation shows that the 
NASDAQ listings do have all the stocks of interest.
Some filtering by sector and removing stocks more recent than the ones used in the course, e.g. TSLA, is
then required to get the tickers of the biggest companies in each sector to match those expected in the course (and 
whose prices are available in the data files).
End of explanation
shares = component_info['Market Capitalization'].div(component_info['Last Sale'])
shares.sort_values(ascending=False)
market_cap_series = data.mul(shares)
_ = market_cap_series.plot(logy=True)
# market_cap_series.first('D').append(market_cap_series.last('D'))  # .append is deprecated
pd.concat([market_cap_series.first('D'), market_cap_series.last('D')], axis=0)
agg_mcap = market_cap_series.sum(axis=1)
_ = agg_mcap.plot(title='Aggregate Market Cap')
mcap_index = agg_mcap.div(agg_mcap.iloc[0]).mul(100)
_ = mcap_index.plot(title='Market-Cap Weighted Index')
Explanation: Build a Market Cap Weighted Index
End of explanation
# Total performance of companies in index
print(f'Companies in index added ${agg_mcap.iloc[-1] - agg_mcap.iloc[0]:,.2f}M in value')
change = pd.concat([market_cap_series.first('D'), market_cap_series.last('D')], axis=0)
change.diff().iloc[-1].sort_values()
weights = component_info['Market Capitalization'].div(component_info['Market Capitalization'].sum())
weights
index_return = (mcap_index.iloc[-1]/mcap_index.iloc[0] - 1)*100
index_return
weighted_return = weights.mul(index_return)
_ = weighted_return.sort_values().plot(kind='barh')
df_index = mcap_index.to_frame('Index')
df_index['SP500'] = pd.read_csv(ddir('sp500.csv'), parse_dates=['date'], index_col='date')
df_index['SP500'] = df_index['SP500'].div(df_index['SP500'].iloc[0]).mul(100)
_ = df_index.plot()
# Multi period returns
def multi_period_returns(r):
    return (np.prod(1 + r) - 1) * 100
_ = df_index.pct_change().rolling('30D').apply(multi_period_returns).plot()
Explanation: Evaluate Index Performance
End of explanation
daily_returns = data.pct_change().dropna()
correlations = daily_returns.corr()
correlations
_ = sns.heatmap(correlations, annot=True, cmap='RdBu', center=0)
_ = plt.xticks(rotation=45)
_ = plt.title('Daily Return Correlations')
# correlations.to_excel(excel_writer=ddir('dc_correlation.xls'), sheet_name='correlations', startrow=1, startcol=1)
with pd.ExcelWriter(ddir('dc_stock_data.xlsx')) as writer:
    correlations.to_excel(excel_writer=writer, sheet_name='correlations')
    data.to_excel(excel_writer=writer, sheet_name='prices')
    data.pct_change().to_excel(excel_writer=writer, sheet_name='returns')
Explanation: Index Correlation and Exporting to Excel
End of explanation
from statsmodels.tsa.arima_process import ArmaProcess
from statsmodels.graphics.tsaplots import plot_acf, plot_pacf
from statsmodels.tsa.stattools import adfuller, coint
from statsmodels.tsa.arima_model import ARMA
daily_returns.head()
len(daily_returns)
_ = plot_acf(daily_returns.AMZN, lags=20, alpha=0.05)
_ = plot_acf((1+daily_returns).resample('Q').prod().AMZN -1 , lags=20, alpha=0.05)
quarterly_returns = data.resample('Q').last().pct_change().dropna()
quarterly_returns.head()
_ = plot_acf(quarterly_returns.AMZN, lags=20, alpha=0.05)
((1 + daily_returns).resample('Q').prod() - 1).head()
# aapl = yf.Ticker('aapl')
# aapl_historical = aapl.history(start="2022-03-01", end="2022-03-05", interval="1m")
# aapl_historical.to_csv(ddir('aapl_data_20220301.csv'))
aapl_historical = pd.read_csv(ddir('aapl_data_20220301.csv'), parse_dates=['Datetime'], index_col='Datetime')
aapl_historical.head()
_ = aapl_historical.loc['2022-03-01', 'Close'].plot()
appl_ret_20220301 = aapl_historical.loc['2022-03-01', 'Close'].pct_change().dropna()
appl_ret_20220301.head()
_ = plot_acf(appl_ret_20220301, lags=60)
_ = aapl_historical.loc['2022-03-01 9:30':'2022-03-01 10:00', 'Close'].plot()
help(coint) #(data.AAPL, data.AMZN)
Explanation: Time Series Analysis
Second course in Skill Track - look at AR, MA, ACF etc
End of explanation
<END_TASK>