Variational Curriculum Reinforcement Learning for Unsupervised Discovery
  of Skills
Seongun Kim Kyowoon Lee Jaesik Choi
Mutual information-based reinforcement learning (RL) has been proposed as a
promising framework for retrieving complex skills autonomously without a
task-oriented reward function through mutual information (MI) maximization or
variational empowerment. However, learning complex skills is still challenging,
due to the fact that the order of training skills can largely affect sample
efficiency. Inspired by this, we recast variational empowerment as curriculum
learning in goal-conditioned RL with an intrinsic reward function, which we
name Variational Curriculum RL (VCRL). From this perspective, we propose a
novel approach to unsupervised skill discovery based on information theory,
called Value Uncertainty Variational Curriculum (VUVC). We prove that, under
regularity conditions, VUVC accelerates the increase of entropy in the visited
states compared to the uniform curriculum. We validate the effectiveness of our
approach on complex navigation and robotic manipulation tasks in terms of
sample efficiency and state coverage speed. We also demonstrate that the skills
discovered by our method successfully complete a real-world robot navigation
task in a zero-shot setup and that incorporating these skills with a global
planner further increases the performance.