Information Content Exploration
Jacob Chmura Hasham Burhani Xiao Qi Shi
Sparse reward environments are known to be challenging for reinforcement
learning agents. In such environments, efficient and scalable exploration is
crucial. Exploration is a means by which an agent gains information about the
environment. We expand on this topic and propose a new intrinsic reward that
systemically quantifies exploratory behavior and promotes state coverage by
maximizing the information content of a trajectory taken by an agent. We
compare our method to alternative exploration based intrinsic reward
techniques, namely Curiosity Driven Learning and Random Network Distillation.
We show that our information theoretic reward induces efficient exploration and
outperforms in various games, including Montezuma Revenge, a known difficult
task for reinforcement learning. Finally, we propose an extension that
maximizes information content in a discretely compressed latent space which
boosts sample efficiency and generalizes to continuous state spaces.