Many practical reinforcement learning problems contain catastrophic states that the optimal policy visits infrequently or never. Even on toy problems, deep reinforcement learners periodically revisit these states, once they are forgotten under a new policy. In this paper, we introduce intrinsic fear, a learned reward shaping that accelerates deep reinforcement learning and guards oscillating policies against periodic catastrophes. Our approach incorporates a second model trained via supervised learning to predict the probability of imminent catastrophe. This score acts as a penalty on the Q-learning objective. Our theoretical analysis demonstrates that the perturbed objective yields the same average return under strong assumptions and an $\epsilon$-close average return under weaker assumptions. Our analysis also shows robustness to classification errors. Equipped with intrinsic fear, our DQNs solve the toy environments and improve on the Atari games Seaquest, Asteroids, and Freeway. Following success on Atari games BID20 and the board game Go BID28 , many researchers have begun exploring practical applications of deep reinforcement learning (DRL). Some investigated applications include robotics BID15 , dialogue systems BID6 BID17 , energy management BID23 , and self-driving cars BID26 . Amid this push to apply DRL, we might ask, can we trust these agents in the wild? Agents acting in real-world environments might possess the ability to cause catastrophic outcomes. Consider a self-driving car that might hit pedestrians or a domestic robot that might injure a child. We might hope to prevent DRL agents from ever making catastrophic mistakes. But doing so requires extensive prior knowledge of the environment in order to constrain the exploration of policy space BID7 .Many conflicting definitions of safety and catastrophe exist, a problem that invites further philosophical consideration. In this paper, we introduce a specific but plausible notion of avoidable catastrophes. These are states that prior knowledge dictates an optimal policy should never visit. For example , we might believe that an optimal self-driving algorithm would never hit a pedestrian. Moreover, we assume that an optimal policy never even comes near an avoidable catastrophe state. We define proximity in trajectory space, and not by the geometry of feature space. We denote states proximal to avoidable catastrophes as danger states. While we don't assume prior knowledge of which states are dangerous, we do assume the existence of a catastrophe detector. After encountering a catastrophic state, an agent can realize this and take action to avoid dangerous states in the future.Given this definition, we address two challenges: First, can we expect DRL agents, after experiencing some number of catastrophic failures, to avoid perpetually making the same mistakes? Second, can we use our prior knowledge that catastrophes should be kept at a distance to accelerate learning of a DRL agent? Our experiments show that even on toy problems, the deep Q-network (DQN), a basic algorithm behind many of today's state-of-the-art DRL systems, struggles on both counts. Even in toy environments , DQNs may encounter thousands of catastrophes before learning to avoid them and are susceptible to repeating old errors. We call this latter problem the Sisyphean curse.This poses a formidable obstacle to using DQNs in the real world. How can we hand over responsibility for consequential actions (control of a car, say) to a DRL agent if it may be doomed to periodically remake every kind of mistake, however grave, so long as it continues to learn? Imagine a self-driving car that had to periodically hit a few pedestrians in order to remember that is undesirable. In the tabular setting, an RL agent never forgets the learned dynamics of its environment, even as its policy evolves. Moreover, if the Markovian assumption holds, eventual convergence to a globally optimal policy is guaranteed. Unfortunately, the tabular approach becomes infeasible in high-dimensional, continuous state spaces.The trouble for DQNs owes to the use of function approximation BID22 . When training a DQN, we successively update a neural network based on experiences. These experiences might be sampled in an online fashion, from a trailing window (experience replay buffer), or uniformly from all past experiences. Regardless of which mode we use to train the network , eventually, states that a learned policy never encounters will come to form an infinitesimally small region of the training distribution. At such times, our networks are subject to the classic problem of catastrophic interference BID19 BID18 . Nothing prevents the DQN's policy from drifting back towards a policy that revisits long-forgotten catastrophic mistakes.More formally, we characterize the problem as unfolding in the following steps: (i) Training under distribution D, our agent produces a safe policy π s that avoids catastrophes (ii) Collecting data generated under π s yields a new distribution of transitions D (iii) Training under D , the agent produces π d , a policy that once again experiences avoidable catastrophes. To illustrate the brittleness of modern DRL algorithms, we introduce a simple pathological problem called Adventure Seeker. This problem consists of a one-dimensional continuous state, two actions , simple dynamics, and a clear analytic solution. Nevertheless, the DQN fails. We then show that similar dynamics exist in the classic RL environment Cart-Pole .In this paper, to combat these problems, we propose intrinsic fear. In this approach, we train a supervised fear model that predicts which states are likely to lead to a catastrophe within k r steps. The output of the fear model (a probability), scaled by a fear factor penalizes the Q-learning target. Our approach draws inspiration from intrinsic motivation BID5 . However, instead of perturbing the reward function to encourage the discovery of novel states, we perturb it to discourage revisiting catastrophic states.We validate the approach both empirically and theoretically. Our experiments address both our Adventure Seeker problem and Cartpole as well as the Atari games Seaquest and Asteroids, and Freeway. For these environments, we label each loss of a life as a catastrophic state. On the toy environments, the intrinsic fear agent learns to avoid death indefinitely , achieving unbounded reward per episode. On Seaquest and Asteroids, the intrinsic fear agent improves markedly and on Freeway the improvement is dramatic. Theoretically, we demonstrate the following: First, we prove that when the reward is bounded and the optimal policy rarely visits the catastrophic states, the policy learned on the altered value function has return similar to the optimal policy on the original value function. Second we prove that the method is robust to noise in the danger model. Our experiments demonstrate that DQNs are susceptible to periodically repeating mistakes, however bad, raising questions about their real-world utility when harm can come of actions. While it's easy to visualize these problems on toy examples, similar dynamics are embedded in more complex domains. Consider a domestic robot acting as a barber. The robot might receive positive feedback for giving a closer shave. This reward encourages closer contact at a steeper angle. Of course, the shape of this reward function belies the catastrophe lurking just past the optimal shave. Similar dynamics might be imagines in a vehicle that is rewarded for traveling faster but could risk an accident with excessive speed. Our results with the intrinsic fear model suggest that with only a small amount of prior knowledge (the ability to recognize catastrophe states after the fact), we can simultaneously accelerate learning and avoid catastrophic states. This work represents a first step towards combating some issues relating to safety in RL stemming from catastrophic forgetting.
Shape reward with intrinsic motivation to avoid catastrophic states and mitigate catastrophic forgetting. An RL algorithm that combines the DQN algorithm with a fear model trained in parallel to predict catastropohic states. The paper studies catastrophic forgetting in RL, by emphasizing tasks where a DQN is able to learn to avoid catastrophic events as long as it avoids forgetting.