Rewards are sparse in the real world and most of today's reinforcement learning algorithms struggle with such sparsity. One solution to this problem is to allow the agent to create rewards for itself - thus making rewards dense and more suitable for learning. In particular, inspired by curious behaviour in animals, observing something novel could be rewarded with a bonus. Such bonus is summed up with the real task reward - making it possible for RL algorithms to learn from the combined reward. We propose a new curiosity method which uses episodic memory to form the novelty bonus. To determine the bonus, the current observation is compared with the observations in memory. Crucially, the comparison is done based on how many environment steps it takes to reach the current observation from those in memory - which incorporates rich information about environment dynamics. This allows us to overcome the known "couch-potato" issues of prior work - when the agent finds a way to instantly gratify itself by exploiting actions which lead to hardly predictable consequences. We test our approach in visually rich 3D environments in ViZDoom, DMLab and MuJoCo. In navigational tasks from ViZDoom and DMLab, our agent outperforms the state-of-the-art curiosity method ICM. In MuJoCo, an ant equipped with our curiosity module learns locomotion out of the first-person-view curiosity only. The code is available at https://github.com/google-research/episodic-curiosity/. Many real-world tasks have sparse rewards. For example, animals searching for food may need to go many miles without any reward from the environment. Standard reinforcement learning algorithms struggle with such tasks because of reliance on simple action entropy maximization as a source of exploration behaviour.Multiple approaches were proposed to achieve better explorative policies. One way is to give a reward bonus which facilitates exploration by rewarding novel observations. The reward bonus is summed up with the original task reward and optimized by standard RL algorithms. Such an approach is motivated by neuroscience studies of animals: an animal has an ability to reward itself for something novel -the mechanism biologically built into its dopamine release system. How exactly this bonus is formed remains an open question.Many modern curiosity formulations aim at maximizing "surprise" -inability to predict the future. This approach makes perfect sense but, in fact, is far from perfect. To show why, let us consider a thought experiment. Imagine an agent is put into a 3D maze. There is a precious goal somewhere in the maze which would give a large reward. Now, the agent is also given a remote control to a TV and can switch the channels. Every switch shows a random image (say, from a fixed set of images). The curiosity formulations which optimize surprise would rejoice because the result of the channel switching action is unpredictable. The agent would be drawn to the TV instead of looking for a goal in the environment (this was indeed observed in BID6 ). So, should we call the channel switching behaviour curious? Maybe, but it is unproductive for the original sparsereward goal-reaching task. What would be a definition of curiosity which does not suffer from such "couch-potato" behaviour?We propose a new curiosity definition based on the following intuition. If the agent knew the observation after changing a TV channel is only one step away from the observation before doing that -it probably would not be so interesting to change the channel in the first place (too easy). This Our method is at the intersection of multiple topics: curiosity, episodic memory and temporal distance prediction. In the following, we discuss the relation to the prior work on those topics. In this work we propose a new model of curiosity based on episodic memory and the ideas of reachability. This allows us to overcome the known "couch-potato" issues of prior work and outperform the previous curiosity state-of-the-art method ICM in visually rich 3D environments from VizDoom and DMLab. Our method also allows a MuJoCo ant to learn locomotion purely out of first-personview curiosity. In the future, we want to make policy aware of memory not only in terms of receiving reward, but also in terms of acting. Can we use memory content retrieved based on reachability to guide exploration behaviour in the test time? This could open opportunities to learn exploration in new tasks in a few-shot style -which is currently a big scientific challenge.
We propose a novel model of curiosity based on episodic memory and the ideas of reachability which allows us to overcome the known "couch-potato" issues of prior work. Proposes to give exploration bonuses in RL algorithms by giving larger bonuses to observations that are father away in environment steps. The authors propose an exploration bonus that is aimed to aid in sparse reward RL problems and considers many experiments on complex 3D environments