Episodic memory is a psychology term which refers to the ability to recall specific events from the past. We suggest one advantage of this particular type of memory is the ability to easily assign credit to a specific state when remembered information is found to be useful. Inspired by this idea, and the increasing popularity of external memory mechanisms to handle long-term dependencies in deep learning systems, we propose a novel algorithm which uses a reservoir sampling procedure to maintain an external memory consisting of a fixed number of past states. The algorithm allows a deep reinforcement learning agent to learn online to preferentially remember those states which are found to be useful to recall later on. Critically this method allows for efficient online computation of gradient estimates with respect to the write process of the external memory. Thus unlike most prior mechanisms for external memory it is feasible to use in an online reinforcement learning setting.
 We present a novel algorithm for integrating a form of external memory with trainable reading and writing into a RL agent. The method depends on the observation that if we restrict the information stored in memory to be a set of past visited states, the information recorded also provides the context in which it was recorded. This means it is possible to assign credit to useful information without needing to backpropagate through time to when it was recorded. To achieve this we devise a reservoir sampling technique which uses a sampling procedure we introduce to generate a distribution over memory configurations for which we can derive gradient estimates. The whole algorithm is O(n) in both the number of trainable parameters and the size of the memory. In particular neither memory required nor computation time increase with history length, making it feasible to run in an online RL setting. We show that the resulting algorithm is able to achieve good performance on a toy problem we introduce designed to have sharp long-term dependencies which can be problematic for recurrent models.
External memory for online reinforcement learning based on estimating gradients over a novel reservoir sampling technique. The paper proposes a modified approach to RL, where an additional "episodic memory" is kept by the agent and use a "query network" that based on the current state.