<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Presented at <a href="http
Step1:  Update PATH_TO_TRAIN and PATH_TO_TEST to the path for rsc15_train_full.txt and rsc15_test.txt respectively
Step2:  Let's take a look to the datasets
Step3:  Sneak Peak to the dataset
Step4:  Training GRU
Step5:  Evaluating GRU
<ASSISTANT_TASK:>
Python Code:
# -*- coding: utf-8 -*-
import theano
import pickle
import sys
import os
sys.path.append('../..')
import numpy as np
import pandas as pd
import gru4rec #If this shows an error probably the notebook is not in GRU4Rec/examples/rsc15/
import evaluation
# Validate that the following assert makes sense in your platform
# This works on Windows with a NVIDIA GPU
# In other platforms theano.config.device gives other things than 'cuda' when using the GPU
assert 'cuda' in theano.config.device,("Theano is not configured to use the GPU. Please check .theanorc. "
                                       "Check http://deeplearning.net/software/theano/tutorial/using_gpu.html")
Explanation: Presented at <a href="http://qvik.fi/"><img style="height:100px" src="https://qvik.com/wp-content/themes/qvik/images/qvik-logo-dark.png"/></a>
Session-based recommender Systems: Hands-on GRU4Rec
Frederick Ayala Gómez, PhD Student in Computer Science at ELTE University. Visiting Researcher at Aalto's Data Mining Group
Let's keep in touch!
Twitter: https://twitter.com/fredayala <br/>
LinkedIn: https://linkedin.com/in/frederickayala <br/>
GitHub: https://github.com/frederickayala
<hr/>
Few notes:
This notebook was tested on Windows and presents how to use GRU4Rec
The paper of GRU4Rec is: B. Hidasi, et al. 2015 “Session-based recommendations with recurrent neural networks”. CoRR
The poster of this paper can be found in http://www.hidasi.eu/content/gru4rec_iclr16_poster.pdf
For OSx and Linux, CUDA, Theano and Anaconda 'might' need some extra steps
On Linux Desktop (e.g. Ubuntu Desktop ), be careful with installing CUDA and NVIDIA drivers. It 'might' break lightdm 🙈🙉🙊
An NVIDIA GEFORCE GTX 980M was used
The starting point of this notebook is the original python demo file from Balázs Hidasi's GRU4REC repository.
It's recommended to use Anaconda to install stuff easier
Installation steps:
Install CUDA 8.0 from https://developer.nvidia.com/cuda-downloads
Optional: Install cuDNN https://developer.nvidia.com/cudnn
Install Anaconda 4.3.1 for Python 3.6 from https://www.continuum.io/downloads
Open Anaconda Navigator
Go to Enviroments / Create / Python Version 3.6 and give some name
In Channels, add: conda-forge then click on Update index...
Click on your enviroment Play arrow and choose Open Terminal
Install the libraries that we need:
conda install numpy scipy pandas mkl-service libpython m2w64-toolchain nose nose-parameterized sphinx pydot-ng
conda install theano pygpu
conda install matplotlib seaborn statsmodels
Create a .theanorc file in your home directory and add the following: <br/>
    [global] <br/>
    device = cuda <br/>
    # Only if you want to use cuDNN <br/>
    [dnn]<br/>
    include_path=/path/to/cuDNN/include <br/>
    library_path=/path/to/cuDNN/lib/x64
Get the GRU4Rec code and the dataset
GRU4Rec:
git clone https://github.com/hidasib/GRU4Rec.git
YOOCHOOSE Dataset:
http://2015.recsyschallenge.com/challenge.html
To get the training and testing files we have to preprocess the original dataset.
Go to the terminal that is running your anaconda enviroment
Navigate to the GRU4Rec folder
Edit the file GRU4Rec/examples/rsc15/preprocess.py and modify the following variables:
PATH_TO_ORIGINAL_DATA The path to the input raw dataset
PATH_TO_PROCESSED_DATA The path to where you want the output
Run the command: python preprocess.py
This will take some time, when the process ends you will have the files rsc15_train_full.txt and rsc15_test.txt in your PATH_TO_PROCESSED_DATA path
Place this notebook in the folder GRU4Rec/examples/rsc15/
That's it! we are ready to run GRU4Rec
End of explanation
PATH_TO_TRAIN = 'C:/Users/frede/datasets/recsys2015/rsc15_train_full.txt'
PATH_TO_TEST = 'C:/Users/frede/datasets/recsys2015/rsc15_test.txt'
data = pd.read_csv(PATH_TO_TRAIN, sep='\t', dtype={'ItemId':np.int64})
valid = pd.read_csv(PATH_TO_TEST, sep='\t', dtype={'ItemId':np.int64})
Explanation: Update PATH_TO_TRAIN and PATH_TO_TEST to the path for rsc15_train_full.txt and rsc15_test.txt respectively
End of explanation
%matplotlib inline
import numpy as np
import pandas as pd
from scipy import stats, integrate
import matplotlib.pyplot as plt
import seaborn as sns
sns.set(color_codes=True)
Explanation: Let's take a look to the datasets
End of explanation
data.head()
valid.head()
sessions_training = set(data.SessionId)
print("There are %i sessions in the training dataset" % len(sessions_training))
sessions_testing = set(valid.SessionId)
print("There are %i sessions in the testing dataset" % len(sessions_testing))
assert len(sessions_testing.intersection(sessions_training)) == 0, ("Huhu!"
                                                                    "there are sessions from the testing set in"
                                                                    "the training set")
print("Sessions in the testing set doesn't exist in the training set")
items_training = set(data.ItemId)
print("There are %i items in the training dataset" % len(items_training))
items_testing = set(valid.ItemId)
print("There are %i items in the testing dataset" % len(items_testing))
assert items_testing.issubset(items_training), ("Huhu!"
                                                "there are items from the testing set "
                                                "that are not in the training set")
print("Items in the testing set exist in the training set")
df_visualization = data.copy()
df_visualization["value"] = 1
df_item_count = df_visualization[["ItemId","value"]].groupby("ItemId").sum()
# Most of the items are infrequent
df_item_count.describe().transpose()
fig = plt.figure(figsize=[15,8])
ax = fig.add_subplot(111)
ax = sns.kdeplot(df_item_count["value"], ax=ax)
ax.set(xlabel='Item Frequency', ylabel='Kernel Density Estimation')
plt.show()
fig = plt.figure(figsize=[15,8])
ax = fig.add_subplot(111)
ax = sns.distplot(df_item_count["value"],
             hist_kws=dict(cumulative=True),
             kde_kws=dict(cumulative=True))
ax.set(xlabel='Item Frequency', ylabel='Cummulative Probability')
plt.show()
# Let's analyze the co-occurrence
df_cooccurrence = data.copy()
df_cooccurrence["next_SessionId"] = df_cooccurrence["SessionId"].shift(-1)
df_cooccurrence["next_ItemId"] = df_cooccurrence["ItemId"].shift(-1)
df_cooccurrence["next_Time"] = df_cooccurrence["Time"].shift(-1)
df_cooccurrence = df_cooccurrence.query("SessionId == next_SessionId").dropna()
df_cooccurrence["next_ItemId"] = df_cooccurrence["next_ItemId"].astype(int)
df_cooccurrence["next_SessionId"] = df_cooccurrence["next_SessionId"].astype(int)
df_cooccurrence.head()
df_cooccurrence["time_difference_minutes"] = np.round((df_cooccurrence["next_Time"] - df_cooccurrence["Time"]) / 60, 2)
df_cooccurrence[["time_difference_minutes"]].describe().transpose()
df_cooccurrence["value"] = 1
df_cooccurrence_sum = df_cooccurrence[["ItemId","next_ItemId","value"]].groupby(["ItemId","next_ItemId"]).sum().reset_index()
df_cooccurrence_sum[["value"]].describe().transpose()
Explanation: Sneak Peak to the dataset
End of explanation
n_layers = 100
save_to = os.path.join(os.path.dirname(PATH_TO_TEST), "gru_" + str(n_layers) +".pickle")
if not os.path.exists(save_to):
    print('Training GRU4Rec with ' + str(n_layers) + ' hidden units')    
    gru = gru4rec.GRU4Rec(layers=[n_layers], loss='top1', batch_size=50, 
                          dropout_p_hidden=0.5, learning_rate=0.01, momentum=0.0)
    gru.fit(data)
    pickle.dump(gru, open(save_to, "wb"))
else:
    print('Loading existing GRU4Rec model with ' + str(n_layers) + ' hidden units')    
    gru = pickle.load(open(save_to, "rb"))
Explanation: Training GRU
End of explanation
res = evaluation.evaluate_sessions_batch(gru, valid, None,cut_off=20)
print('The proportion of cases having the desired item within the top 20 (i.e Recall@20): {}'.format(res[0]))
batch_size = 500
print("Now let's try to predict over the first %i items of our testint dataset" % batch_size)
df_valid = valid.head(batch_size)
df_valid["next_ItemId"] = df_valid["ItemId"].shift(-1)
df_valid["next_SessionId"] = df_valid["SessionId"].shift(-1)
session_ids = valid.head(batch_size)["SessionId"].values
input_item_ids = valid.head(batch_size)["ItemId"].values
predict_for_item_ids=None
%timeit gru.predict_next_batch(session_ids=session_ids, input_item_ids=input_item_ids, predict_for_item_ids=None, batch=batch_size)
df_preds = gru.predict_next_batch(session_ids=session_ids, 
                      input_item_ids=input_item_ids,
                      predict_for_item_ids=None,
                      batch=batch_size)
df_valid.shape
df_preds.shape
df_preds.columns = df_valid.index.values
len(items_training)
df_preds
for c in df_preds:
    df_preds[c] = df_preds[c].rank(ascending=False)
df_valid_preds = df_valid.join(df_preds.transpose())
df_valid_preds = df_valid_preds.query("SessionId == next_SessionId").dropna()
df_valid_preds["next_ItemId"] = df_valid_preds["next_ItemId"].astype(int)
df_valid_preds["next_SessionId"] = df_valid_preds["next_SessionId"].astype(int)
df_valid_preds["next_ItemId_at"] = df_valid_preds.apply(lambda x: x[int(x["next_ItemId"])], axis=1)
df_valid_preds_summary = df_valid_preds[["SessionId","ItemId","Time","next_ItemId","next_ItemId_at"]]
df_valid_preds_summary.head(20)
cutoff = 20
df_valid_preds_summary_ok = df_valid_preds_summary.query("next_ItemId_at <= @cutoff")
df_valid_preds_summary_ok.head(20)
recall_at_k = df_valid_preds_summary_ok.shape[0] / df_valid_preds_summary.shape[0]
print("The recall@%i for this batch is %f"%(cutoff,recall_at_k))
fig = plt.figure(figsize=[15,8])
ax = fig.add_subplot(111)
ax = sns.kdeplot(df_valid_preds_summary["next_ItemId_at"], ax=ax)
ax.set(xlabel='Next Desired Item @K', ylabel='Kernel Density Estimation')
plt.show()
fig = plt.figure(figsize=[15,8])
ax = fig.add_subplot(111)
ax = sns.distplot(df_valid_preds_summary["next_ItemId_at"],
             hist_kws=dict(cumulative=True),
             kde_kws=dict(cumulative=True))
ax.set(xlabel='Next Desired Item @K', ylabel='Cummulative Probability')
plt.show()
print("Statistics for the rank of the next desired item (Lower the best)")
df_valid_preds_summary[["next_ItemId_at"]].describe()
Explanation: Evaluating GRU
End of explanation
<END_TASK>