Sparse reward is one of the most challenging problems in reinforcement learning (RL). Hindsight Experience Replay (HER) attempts to address this issue by converting a failure experience to a successful one by relabeling the goals. Despite its effectiveness, HER has limited applicability because it lacks a compact and universal goal representation. We present Augmenting experienCe via TeacheR's adviCE (ACTRCE), an efficient reinforcement learning technique that extends the HER framework using natural language as the goal representation. We first analyze the differences among goal representation, and show that ACTRCE can efficiently solve difficult reinforcement learning problems in challenging 3D navigation tasks, whereas HER with non-language goal representation failed to learn. We also show that with language goal representations, the agent can generalize to unseen instructions, and even generalize to instructions with unseen lexicons. We further demonstrate it is crucial to use hindsight advice to solve challenging tasks, but we also found that little amount of hindsight advice is sufficient for the learning to take off, showing the practical aspect of the method. Many impressive deep reinforcement learning (deep RL) applications rely on carefully-crafted reward functions to encourage the desired behavior. However, designing a good reward function is nontrivial (Ng et al., 1999) , and requires a significant engineering effort. For example, even for the seemingly simple task of stacking Lego blocks, Popov et al. (2017) needed 5 complicated reward terms with different importance weights. Moreover, handcrafted reward shaping BID9 can lead to biased learning, which may cause the agent to learn unexpected and undesired behaviors BID6 .One approach to avoid defining a complicated reward function is to use a sparse and binary reward function, i.e., give only a positive or negative reward at the terminal state, depending on the success of the task. However , the sparse reward makes learning difficult.Hindsight Experience Replay (HER) BID1 attempts to address this issue. The main idea of HER is to utilize failed experiences by substituting with a fake goal in order to convert them to successful experiences. For their algorithm to work, Andrychowicz et al. made the non-trivial assumption that for every state in the environment, there exists a goal which is achieved in that state. As the authors pointed out, this assumption can be trivially satisfied by choosing the goal space to be the state space. However, representing the goal using the enormous state space is very inefficient and contains much redundant information. For example, if we want to ask an agent to avoid collisions while driving where the state is the raw pixel value from the camera, then there can be many states (i.e. frames) that achieve this goal. It is redundant to represent the goal using each state.Therefore, we need a goal representation that is (1) expressive and flexible enough to satisfy the assumption in HER, while also being (2) compact and informative where similar goals are represented using similar features. Natural language representation of goals satisfies both requirements. First, language can flexibly describe goals across tasks and environments. Second, language representation is abstract , hence able to compress any redundant information in the states. Recall the previous driving example, for which we can simply describe "avoid collisions" to represent all states that satisfy this goal. Moreover, the compositional nature of language provides transferable features for generalizing across goals.In this paper, we combine the HER framework with natural language goal representation, and propose an efficient technique called Augmenting experienCe via TeacheR's adviCE (ACTRCE; pronounced "actress") to a broad range of reinforcement learning problems. Our method works as follows. Whenever an agent finishes an episode, a teacher gives advice in natural language to the agent based on the episode. The agent takes the advice to form a new experience with a corresponding reward, alleviating the sparse reward problem. For example, a teacher can describe what the agent has achieved in the episode, and the agent can replace the original goal with the advice and a reward of 1. We show many benefits brought by language goal representation when combining with hindsight advice. The agent can efficiently solve reinforcement learning problems in challenging 2D and 3D environments; it can generalize to unseen instructions, and even generalize to instruction with unseen lexicons. We further demonstrate it is crucial to use hindsight advice to solve challenging tasks, but we also found that little amount of hindsight advice is sufficient for the learning to take off, showing the practical aspect of the method.We note that our work is also interesting from a language learning perspective. Learning to achieve goals described in natural language is part of a class of problem called language grounding BID10 , which has recently received increasing interest, as grounding is believed to be necessary for more general understanding of natural language. Early attempts to ground language in a simulated physical world (Winograd, 1972; Siskind, 1994) consisted of hard coded rules which could not scale beyond their original domain. Recent work has been using reinforcement learning techniques to address this problem BID12 BID4 . Our work combines reinforcement learning and rich language advice , providing an efficient technique for language grounding. In this work, we propose the ACTRCE method that uses natural language as a goal representation for hindsight advice. The main point of the paper is to show that using language as goal representations can bring many benefits, when combined with hindsight advice. We analyzed the differences among goal representation, and show that ACTRCE can efficiently solve difficult reinforcement learning problems in challenging 3D navigation tasks, whereas HER with non-language goal representation failed to learn. We also show that with language goal representations, the agent can generalize to unseen instructions. With pre-trained language component, the agent can even generalize to instructions with unseen lexicons, demonstrating its potential to deal with noisy natural language advice from humans. Although ACTRCE algorithm crucially relies on hindsight advice, we showed that little amount of advice is sufficient for the learning to take off, showing its great practicality.
Combine language goal representation with hindsight experience replays. This paper considers the assumption implicit in hindsight experience replay, that there is access to a mapping from states to goals, and proposes a natural language goal representation. This submission uses Hindsight Experience Replay framework with natural language goals to improve the sample-efficiency of instruction-following models.