Acquiring Diverse Skills using Curriculum Reinforcement Learning with
  Mixture of Experts
Onur Celik Aleksandar Taranovic Gerhard Neumann
Reinforcement learning (RL) is a powerful approach for acquiring a
good-performing policy. However, learning diverse skills is challenging in RL
due to the commonly used Gaussian policy parameterization. We propose
\textbf{Di}verse \textbf{Skil}l \textbf{L}earning (Di-SkilL), an RL method for
learning diverse skills using Mixture of Experts, where each expert formalizes
a skill as a contextual motion primitive. Di-SkilL optimizes each expert and
its associate context distribution to a maximum entropy objective that
incentivizes learning diverse skills in similar contexts. The per-expert
context distribution enables automatic curricula learning, allowing each expert
to focus on its best-performing sub-region of the context space. To overcome
hard discontinuities and multi-modalities without any prior knowledge of the
environment's unknown context probability space, we leverage energy-based
models to represent the per-expert context distributions and demonstrate how we
can efficiently train them using the standard policy gradient objective. We
show on challenging robot simulation tasks that Di-SkilL can learn diverse and
performant skills.