This paper investigates whether learning contingency-awareness and controllable aspects of an environment can lead to better exploration in reinforcement learning. To investigate this question, we consider an instantiation of this hypothesis evaluated on the Arcade Learning Element (ALE). In this study, we develop an attentive dynamics model (ADM) that discovers controllable elements of the observations, which are often associated with the location of the character in Atari games. The ADM is trained in a self-supervised fashion to predict the actions taken by the agent. The learned contingency information is used as a part of the state representation for exploration purposes. We demonstrate that combining actor-critic algorithm with count-based exploration using our representation achieves impressive results on a set of notoriously challenging Atari games due to sparse rewards. For example, we report a state-of-the-art score of >11,000 points on Montezuma's Revenge without using expert demonstrations, explicit high-level information (e.g., RAM states), or supervisory data. Our experiments confirm that contingency-awareness is indeed an extremely powerful concept for tackling exploration problems in reinforcement learning and opens up interesting research questions for further investigations. The success of reinforcement learning (RL) algorithms in complex environments hinges on the way they balance exploration and exploitation. There has been a surge of recent interest in developing effective exploration strategies for problems with high-dimensional state spaces and sparse rewards BID8 Oudeyer & Kaplan, 2009; Houthooft et al., 2016; Bellemare et al., 2016; Osband et al., 2016; Pathak et al., 2017; BID5 BID16 . Deep neural networks have seen great success as expressive function approximators within RL and as powerful representation learning methods for many domains. In addition, there have been recent studies on using neural network representations for exploration BID13 Martin et al., 2017; Pathak et al., 2017) . For example, count-based exploration with neural density estimation (Bellemare et al., 2016; BID13 Ostrovski et al., 2017) presents one of the state-of-the-art techniques on the most challenging Atari games with sparse rewards.Despite the success of recent exploration methods, it is still an open question on how to construct an optimal representation for exploration. For example, the concept of visual similarity is used for learning density models as a basis for calculating pseudo-counts (Bellemare et al., 2016; Ostrovski et al., 2017) . However, as BID13 noted, the ideal way to represent states should be based on what is relevant to solving the MDP, rather than only relying on visual similarity. In addition, there remains another question on whether the representations used for recent exploration works are easily interpretable. To address these questions, we investigate whether we can learn a complementary, more intuitive, and interpretable high-level abstraction that can be very effective in exploration by using the ideas of contingency awareness and controllable dynamics.The key idea that we focus on in this work is the notion of contingency awareness BID14 Bellemare et al., 2012) -the agent's understanding of the environmental dynamics and recognizing that some aspects of the dynamics are under the agent's control. Intuitively speaking, this can represent the segmentation mask of the agent operating in the 2D or 3D environments (yet one can think of more abstract and general state spaces). In this study, we investigate the concept of contingency awareness based on self-localization, i.e., the awareness of where the agent is located in the abstract state space. We are interested in discovering parts of the world that are directly dependent on the agent's immediate action, which often reveal the agent's approximate location.For further motivation on the problem, we note that contingency awareness is a very important concept in neuroscience and psychology. In other words, being self-aware of one's location is an important property within many observed intelligent organisms and systems. For example, recent breakthroughs in neuroscience, such as the Nobel Prize winning work on the grid cells (Moser et al., 2015; BID4 , show that organisms that perform very well in spatially-challenging tasks are self-aware of their location. This allows rats to navigate, remember paths to previously visited places and important sub-goals, and find shortcuts. In addition, the notion of contingency awareness has been shown as an important factor in developmental psychology BID14 BID2 . We can think of self-localization (and more broadly self-awareness) as a principled and fundamental direction towards intelligent agents.Based on these discussions, we hypothesize that contingency awareness can be a powerful mechanism for tackling exploration problems in reinforcement learning. We consider an instantiation of this hypothesis evaluated on the Arcade Learning Element (ALE). For example, in the context of 2D Atari games, contingency-awareness roughly corresponds to understanding the notion of controllable entities (e.g., the player's avatar), which Bellemare et al. (2012) refer to as contingent regions. More concretely, as shown in FIG0 , in the game FREEWAY, only the chicken sprite is under the agent's control and not the multiple moving cars; therefore the chicken's location should be an informative element for exploration (Bellemare et al., 2012; Pathak et al., 2017) .In this study, we also investigate whether contingency awareness can be learned without any external annotations or supervision. For this, we provide an instantiation of an algorithm for automatically learning such information and using it for improving exploration on a 2D ALE environment (Bellemare et al., 2013) . Concretely , we employ an attentive dynamics model (ADM) to predict the agent's action chosen between consecutive states. It allows us to approximate the agent's position in 2D environments, but unlike other approaches such as (Bellemare et al., 2012) , it does not require any additional supervision to do so. The ADM learns in an online and self-supervised fashion with pure observations as the agent's policy is updated and does not require hand-crafted features, an environment simulator, or supervision labels for training.In experimental evaluation, our methods significantly improve the performance of A2C on hardexploration Atari games in comparison with competitive methods such as density-based exploration (Bellemare et al., 2016; Ostrovski et al., 2017) and SimHash BID13 . We report very strong results on sparse-reward Atari games, including the state-of-the-art performance on the notoriously difficult MONTEZUMA'S REVENGE, when combining our proposed exploration strategy with PPO , without using expert demonstrations, explicit high-level information (e.g., RAM states), or resetting the environment to an arbitrary state.We summarize our contributions as follows:• We demonstrate the importance of learning contingency awareness for efficient exploration in challenging sparse-reward RL problems.• We develop a novel instance of attentive dynamics model using contingency and controllable dynamics to provide robust localization abilities across the most challenging Atari environments.• We achieve a strong performance on difficult sparse-reward Atari games, including the state-ofthe-art score on the notoriously challenging MONTEZUMA'S REVENGE.Overall, we believe that our experiments confirm the hypothesis that contingency awareness is an extremely powerful concept for tackling exploration problems in reinforcement learning, which opens up interesting research questions for further investigations. This paper investigates whether discovering controllable dynamics via an attentive dynamics model (ADM) can help exploration in challenging sparse-reward environments. We demonstrate the effectiveness of this approach by achieving significant improvements on notoriously difficult video games. That being said, we acknowledge that our approach has certain limitations. Our currently presented instance of state abstraction method mainly focuses on controllable dynamics and employs a simple clustering scheme to abstract away uncontrollable elements of the scene. In more general setting, one can imagine using attentive (forward or inverse) dynamics models to learn an effective and compact abstraction of the controllable and uncontrollable dynamics as well, but we leave this to future work.Key elements of the current ADM method include the use of spatial attention and modelling of the dynamics. These ideas can be generalized by a set of attention-based dynamics models (ADM) operating in forward, inverse, or combined mode. Such models could use attention over a lowerdimensional embedding that corresponds to an intrinsic manifold structure from the environment (i.e., intuitively speaking, this also corresponds to being self-aware of (e.g., locating) where the agent is in the abstract state space). Our experiments with the inverse dynamics model suggest that the mechanism does not have to be perfectly precise, allowing for some error in practice. We speculate that mapping to such subspace could be obtained by techniques of embedding learning.We note that RL environments with different visual characteristics may require different forms of attention-based techniques and properties of the model (e.g., partial observability). Even though this paper focuses on 2D video games, we believe that the presented high-level ideas of learning contingency-awareness (with attention and dynamics models) are more general and could be applicable to more complex 3D environments with some extension. We leave this as future work. We proposed a method of providing contingency-awareness through an attentive dynamics model (ADM). It enables approximate self-localization for an RL agent in 2D environments (as a specific perspective). The agent is able to estimate its position in the space and therefore benefits from a compact and informative representation of the world. This idea combined with a variant of countbased exploration achieves strong results in various sparse-reward Atari games. Furthermore, we report state-of-the-art results of >11,000 points on the infamously challenging MONTEZUMA'S REVENGE without using expert demonstrations or supervision. Though in this work we focus mostly on 2D environments in the form of sparse-reward Atari games, we view our presented high-level concept and approach as a stepping stone towards more universal algorithms capable of similar abilities in various RL environments. DISPLAYFORM0 Perform actor-critic using on-policy samples in E θ A2C ← θ A2C − η∇ θA2C L
We investigate contingency-awareness and controllable aspects in exploration and achieve state-of-the-art performance on Montezuma's Revenge without expert demonstrations. This paper investigates the problem of extracting a meaningful state representation to help with exploration when confronted with a sparse reward task by identifying controllable (learned) features of the state This paper proposes the novel idea of using contingency awareness to aid exploration in sparse-reward reinforcement learning tasks, obtaining state of the art results.