<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 JPX Tokyo Stock Exchange Original Notebook
In this Kaggle competition 
Japan Exchange Group, Inc. (JPX) is a holding company operating one of the largest stock exchanges in the world, Tokyo Stock Exchange (TSE), and derivatives exchanges Osaka Exchange (OSE) and Tokyo Commodity Exchange (TOCOM). JPX is hosting this competition and is supported by AI technology company AlpacaJapan Co.,Ltd.
In this competition, you will model real future returns of around 2,000 stocks. The competition will involve building portfolios from the stocks eligible for predictions. The stocks are ranked from highest to lowest expected returns and they are evaluated on the difference in returns between the top and bottom 200 stocks.
Step1:  Imports
In this section we import the packages we need for this example. Make it a habit to gather your imports in a single place.
Step2:  Project hyper-parameters
Step3:  Set random seed for reproducibility and ignore warning messages.
Step4:  Download and load the dataset
In this section, we download the data from kaggle to get it in a ready-to-use form by the model. 
First, let us load and analyze the data.
The data are in csv format, thus, we use the handy read_csv pandas method. There is one train data set and two test sets (one public and one private).
Step5:  Transform Data
Step6:  <h1>Feature Engineering
Step7:  Modelling
Step8:  <h1> Evaluation and Prediction
Step9:  Make submission
<ASSISTANT_TASK:>
Python Code:
!pip install -r requirements.txt --user --quiet
Explanation: JPX Tokyo Stock Exchange Original Notebook
In this Kaggle competition 
Japan Exchange Group, Inc. (JPX) is a holding company operating one of the largest stock exchanges in the world, Tokyo Stock Exchange (TSE), and derivatives exchanges Osaka Exchange (OSE) and Tokyo Commodity Exchange (TOCOM). JPX is hosting this competition and is supported by AI technology company AlpacaJapan Co.,Ltd.
In this competition, you will model real future returns of around 2,000 stocks. The competition will involve building portfolios from the stocks eligible for predictions. The stocks are ranked from highest to lowest expected returns and they are evaluated on the difference in returns between the top and bottom 200 stocks.
End of explanation
import sys, os
from tqdm import tqdm
import subprocess
import numpy as np
import pandas as pd
from scipy import stats
import matplotlib.pyplot as plt
import zipfile
import joblib
import gc
from lightgbm import LGBMRegressor
from sklearn.metrics import mean_squared_error
from sklearn.model_selection._split import _BaseKFold, indexable, _num_samples
from sklearn.utils.validation import _deprecate_positional_args
pd.set_option('display.max_columns', 500)
sys.path.insert(0, 'helper-files')
from local_api import local_api
Explanation: Imports
In this section we import the packages we need for this example. Make it a habit to gather your imports in a single place.
End of explanation
# Hyper-parameters
LR = 0.379687157316759
N_EST = 100
Explanation: Project hyper-parameters
End of explanation
np.random.seed(42)
Explanation: Set random seed for reproducibility and ignore warning messages.
End of explanation
# setup kaggle environment for data download
# set kaggle.json path
os.environ['KAGGLE_CONFIG_DIR'] = "/home/jovyan/examples/jpx-tokyo-stock-exchange-kaggle-competition"
# grant rwo permission to .kaggle/kaggle.json
!chmod 600 /home/jovyan/examples/jpx-tokyo-stock-exchange-kaggle-competition/kaggle.json
# download kaggle's jpx-tokyo-stock-exchange-prediction data
!kaggle competitions download -c jpx-tokyo-stock-exchange-prediction
# path to download to
data_path = os.getcwd()
# extract jpx-tokyo-stock-exchange-prediction.zip to load_data_path
with zipfile.ZipFile("jpx-tokyo-stock-exchange-prediction.zip","r") as zip_ref:
    zip_ref.extractall(data_path)
# read train_files/stock_prices.csv
df_prices = pd.read_csv(f"{data_path}/train_files/stock_prices.csv", parse_dates=['Date'])
df_prices['Date'].max()
df_prices.tail(3)
# lets check data dimensions
df_prices.shape
df_prices.info()
# check total nan values per column
df_prices.isna().sum()
Explanation: Download and load the dataset
In this section, we download the data from kaggle to get it in a ready-to-use form by the model. 
First, let us load and analyze the data.
The data are in csv format, thus, we use the handy read_csv pandas method. There is one train data set and two test sets (one public and one private).
End of explanation
# sort data by 'Date' and 'SecuritiesCode'
df_prices.sort_values(by=['Date','SecuritiesCode'], inplace=True)
# sort data by 'Date' and 'SecuritiesCode'
df_prices.sort_values(by=['Date','SecuritiesCode'], inplace=True)
# count total trading stocks per day 
idcount = df_prices.groupby("Date")["SecuritiesCode"].count().reset_index()
idcount
plt.figure(figsize=(10, 5))
plt.plot(idcount["Date"],idcount["SecuritiesCode"])
plt.axvline(x=['2021-01-01'], color='blue', label='2021-01-01')
plt.axvline(x=['2020-06-01'], color='red', label='2020-06-01')
plt.legend()
plt.show()
idcount[idcount['SecuritiesCode'] >= 2000]
idcount[idcount['SecuritiesCode'] >= 2000]['SecuritiesCode'].sum()
# filter out data with less than 2000 stock counts in a day
# dates before ‘2020-12-23’ all have stock counts less than 2000
# This is done to work with consistent data 
df_prices = df_prices[(df_prices["Date"]>="2020-12-23")]
df_prices = df_prices.reset_index(drop=True)
df_prices.head()
df_prices.columns
#calculate z-scores of `df`
z_scores = stats.zscore(df_prices[['Open', 'High', 'Low', 'Close','Volume']], nan_policy='omit')
abs_z_scores = np.abs(z_scores)
filtered_entries = (abs_z_scores < 3).all(axis=1)
df_zscore = df_prices[filtered_entries]
df_zscore = df_zscore.reset_index(drop=True)
df_zscore = df_zscore.reset_index(drop=True)
Explanation: Transform Data
End of explanation
def feat_eng(df, features):
    for i in tqdm(range(1, 4)):
        # creating lag features
        tmp = df[features].shift(i)
        tmp.columns = [c + f'_next_shift_{i}' for c in tmp.columns]
        df = pd.concat([df, tmp], sort=False, axis=1)
    for i in tqdm(range(1, 4)):
        df[f'weighted_vol_price_{i}'] = np.log(df[f'Volume_next_shift_{i}'] * df[[col for col in df if col.endswith(f'next_shift_{i}')][:-1]].apply(np.mean, axis=1))
       
    # feature engineering
    df['weighted_vol_price'] = np.log(df['Volume'] * (np.mean(df[features[:-1]], axis=1)))
    df['BOP'] = (df['Open']-df['Close'])/(df['High']-df['Low'])
    df['HL'] = df['High'] - df['Low']
    df['OC'] = df['Close'] - df['Open']
    df['OHLCstd'] = df[['Open','Close','High','Low']].std(axis=1)
    
    feats = df.select_dtypes(include=float).columns
    df[feats] = df[feats].apply(np.log)
    
    # replace inf with nan
    df.replace([np.inf, -np.inf], np.nan, inplace=True)
    
    # datetime features
    df['Date'] = pd.to_datetime(df['Date'])
    df['Day'] = df['Date'].dt.weekday.astype(np.int32)
    df["dayofyear"] = df['Date'].dt.dayofyear
    df["is_weekend"] = df['Day'].isin([5, 6])
    df["weekofyear"] = df['Date'].dt.weekofyear
    df["month"] = df['Date'].dt.month
    df["season"] = (df["month"]%12 + 3)//3
    
    # fill nan values
    df = df.fillna(0)
    return df
new_feats = feat_eng(df_zscore, ['High', 'Low', 'Open', 'Close', 'Volume'])
new_feats.shape
new_feats['Target'] = df_zscore['Target']
new_feats.head(7)
new_feats.columns
Explanation: <h1>Feature Engineering
End of explanation
# columns to be used for modelling.
feats = ['Date','SecuritiesCode', 'Open', 'High', 'Low', 'Close', 'Volume',
         'weighted_vol_price_1', 'weighted_vol_price_2', 'weighted_vol_price_3', 
       'weighted_vol_price', 'BOP', 'HL', 'OC', 'OHLCstd', 'Day', 'dayofyear',
       'is_weekend', 'weekofyear', 'month', 'season']
# transform date to int
new_feats['Date'] = new_feats['Date'].dt.strftime("%Y%m%d").astype(int)
# split data into valid for validation and train for model training
valid = new_feats[(new_feats['Date'] >= 20211111)].copy()
train = new_feats[(new_feats['Date'] < 20211111)].copy()
train.shape, valid.shape
# model parameter
params = {
  'n_estimators': N_EST,
  'learning_rate': LR,
    'random_state': 42,
  'verbose' : 2}
# model initialization
model = LGBMRegressor(**params)
X = train[feats]
y = train["Target"]
X_test = valid[feats]
y_test = valid["Target"]
# fitting
model.fit(X, y, verbose=False, eval_set=(X_test, y_test))
Explanation: Modelling
End of explanation
# model prediction
preds = model.predict(X_test)
# model evaluation
rmse = np.round(mean_squared_error(preds, y_test)**0.5, 5)
print(rmse)
Explanation: <h1> Evaluation and Prediction
End of explanation
myapi = local_api('data/supplemental_files')
env = myapi.make_env()
iter_test = env.iter_test()
for (prices, options, financials, trades, secondary_prices, sample_prediction) in iter_test:
    prices = feat_eng(prices, ['High', 'Low', 'Open', 'Close', 'Volume'])
    prices['Date'] = prices['Date'].dt.strftime("%Y%m%d").astype(int)
    prices["Target"] = model.predict(prices[feats])
    if prices["Volume"].min()==0:
        sample_prediction["Prediction"] = 0
    else:
        sample_prediction["Prediction"] = prices["Target"]/prices["Volume"]
    sample_prediction["Prediction"] = prices["Target"]
    sample_prediction.sort_values(by="Prediction", ascending=False, inplace=True)
    sample_prediction['Rank'] = np.arange(0,2000)
    sample_prediction.sort_values(by = "SecuritiesCode", ascending=True, inplace=True)
    submission = sample_prediction[["Date","SecuritiesCode","Rank"]]
    env.predict(submission)
print(env.score())
submission.head()
Explanation: Make submission
End of explanation
<END_TASK>