One of the key challenges of session-based recommender systems is to enhance users’ purchase intentions. In this paper, we formulate the sequential interactions between user sessions and a recommender agent as a Markov Decision Process (MDP). In practice, the purchase reward is delayed and sparse, and may be buried by clicks, making it an impoverished signal for policy learning. Inspired by the prediction error minimization (PEM) and embodied cognition, we propose a simple architecture to augment reward, namely Imagination Reconstruction Network (IRN). Speciﬁcally, IRN enables the agent to explore its environment and learn predictive representations via three key components. The imagination core generates predicted trajectories, i.e., imagined items that users may purchase. The trajectory manager controls the granularity of imagined trajectories using the planning strategies, which balances the long-term rewards and short-term rewards. To optimize the action policy, the imagination-augmented executor minimizes the intrinsic imagination error of simulated trajectories by self-supervised reconstruction, while maximizing the extrinsic reward using model-free algorithms. Empirically, IRN promotes quicker adaptation to user interest, and shows improved robustness to the cold-start scenario and ultimately higher purchase performance compared to several baselines. Somewhat surprisingly, IRN using only the purchase reward achieves excellent next-click prediction performance, demonstrating that the agent can "guess what you like" via internal planning. A good recommender system can enhance both satisfaction for users and profit for content providers BID7 . In many real-world scenarios, the recommender systems make recommendations based only on the current browsing session, given the absence of user profiles (because the user is new or not tracked or not logged in, till the final purchase step). A session is a group of sequential interactions between a user and the system within a short period of time. To model this phenomenon, Recurrent Neural Networks (RNNs) were recently employed as session-based recommenders BID9 BID12 . For instance, GRU4Rec BID9 utilizes the session-parallel mini-batch training to handle the variable lengths of sessions, and predicts the next action given the sequence of items in the current session. However, these approaches primarily focus on next-click prediction and model the session data via sequential classification, and thus cannot distinguish the different effects of user clicks and purchases.In this paper, we consider the session-based recommendation as a Markov Decision Process (MDP), which can take into account both the click reward and the purchase reward (see FIG0 , and leverage Reinforcement Learning (RL) to learn the recommendation strategy. In practice, several challenges need to be addressed. First, the recommender systems involve large numbers of discrete actions (i.e., items), making current RL algorithms difficult to apply . This requires the agent to explore its environment for action feature learning and develop an ability to generalize over unseen actions. Second, we found it difficult to specify the click reward and the purchase reward; the policy may be biased by long sessions that contain many user clicks, as RL algorithms maximize the accumulated reward. Besides, real-world recommender systems require quick adaptation to user interest and robustness to the cold-start scenario (i.e., enhancing the purchase performance of short sessions). Therefore, we will be particularly interested in a case where only the purchase is used as reward (click sequences are used as inputs of the imagination core for Under review as a conference paper at ICLR 2019 exploration). 1 However, the purchase reward is delayed and sparse (one session may contain only one purchase), making it a difficult signal for policy learning.To augment reward and encourage exploration, we present the Imagination Reconstruction Network (IRN), which is inspired by the prediction error minimization (PEM) BID10 BID5 BID14 and embodied cognition BID2 BID1 BID23 BID3 from the neuroscience literature. The PEM is an increasingly influential theory that stresses the importance of brain-body-world interactions in cognitive processes, involving perception, action and learning. In particular, IRN can be regarded as a proof-of-concept for the PEM from the recommendation perspective, following the ideas in BID1 and BID23 -the brain utilizes active sensorimotor predictions (or counterfactual predictions) to represent states of affairs in the world in an action-oriented manner. Specifically, the imagination core of IRN that predicts the future trajectories (i.e., a set of imagined items that user may purchase) conditioned on actions sampled from the imagination policy, can be considered as the generative model of the brain that simulates sensorimotor predictions. To update the action policy, the imagination-augmented executor minimizes the intrinsic imagination error of predicted trajectories by self-supervised reconstruction, while maximizing the extrinsic reward using RL, with shared input state or output action representations for predictive learning. This simulates the active perception (a key aspect of embodied cognition) of the body under the PEM framework, which adapts the agent to possible changes that arise from the ongoing exploratory action. Note that the imagination policy imitates the action policy through distillation or a delayed target network, and thus IRN constructs a loop between brain and body, encouraging the agent to perform actions that can reduce the error in the agent's ability to predict the future events BID20 . IRN equips the agent with a planning module, trajectory manager, that controls the granularity of imagined trajectories using the planning strategies (e.g., breadth-n and depth-m). Besides, IRN is a combination of model-based planning and self-supervised RL, as the imagined trajectories provide dense training signals for auxiliary task learning (see section 2).The key contributions of this paper are summarized as follows:• We formulate the session-based recommendation as a MDP, and leverage deep RL to learn the optimal recommendation policy, and also discuss several challenges when RL is applied.• We consider a special case where only the purchase is used as reward, and then propose the IRN architecture to optimize the sparser but more business-critical purchase signals, which draws inspiration from the theories of cognition science.• We present a self-supervised reconstruction method for predictive learning, which minimizes the imagination error of simulated trajectories over time. IRN achieves excellent click and purchase performance even without any external reward (predictive perception BID23 ).• We conduct a comprehensive set of experiments to demonstrate the effectiveness of IRN. Compared to several baselines, IRN improves data efficiency, promotes quicker adaptation to user interest, and shows improved robustness to the cold-start scenario and ultimately higher purchase performance. These are highly valuable properties in an industrial context.
We propose the IRN architecture to augment sparse and delayed purchase reward for session-based recommendation. The paper proposes improving the performance of recommendation systems through reinforcement learning by using an Imagination Reconstruction Network. The paper presents a session-based recommendation approach by focusing on user purchases instead of clicks. 