First-Explore, then Exploit: Meta-Learning Intelligent Exploration
Ben Norman Jeff Clune
Standard reinforcement learning (RL) agents never intelligently explore like
a human (i.e. by taking into account complex domain priors and previous
explorations). Even the most basic intelligent exploration strategies such as
exhaustive search are only inefficiently or poorly approximated by approaches
such as novelty search or intrinsic motivation, let alone more complicated
strategies like learning new skills, climbing stairs, opening doors, or
conducting experiments. This lack of intelligent exploration limits sample
efficiency and prevents solving hard exploration domains. We argue a core
barrier prohibiting many RL approaches from learning intelligent exploration is
that the methods attempt to explore and exploit simultaneously, which harms
both exploration and exploitation as the goals often conflict. We propose a
novel meta-RL framework (First-Explore) with two policies: one policy learns to
only explore and one policy learns to only exploit. Once trained, we can then
explore with the explore policy, for as long as desired, and then exploit based
on all the information gained during exploration. This approach avoids the
conflict of trying to do both exploration and exploitation at once. We
demonstrate that First-Explore can learn intelligent exploration strategies
such as exhaustive search and more, and that it outperforms dominant standard
RL and meta-RL approaches on domains where exploration requires sacrificing
reward. First-Explore is a significant step towards creating meta-RL algorithms
capable of learning human-level exploration which is essential to solve
challenging unseen hard-exploration domains.