Synergizing Quality-Diversity with Descriptor-Conditioned Reinforcement
  Learning
Maxence Faldor Félix Chalumeau Manon Flageat Antoine Cully
A fundamental trait of intelligence involves finding novel and creative
solutions to address a given challenge or to adapt to unforeseen situations.
Reflecting this, Quality-Diversity optimization is a family of Evolutionary
Algorithms, that generates collections of both diverse and high-performing
solutions. Among these, MAP-Elites is a prominent example, that has been
successfully applied to a variety of domains, including evolutionary robotics.
However, MAP-Elites performs a divergent search with random mutations
originating from Genetic Algorithms, and thus, is limited to evolving
populations of low-dimensional solutions. PGA-MAP-Elites overcomes this
limitation using a gradient-based variation operator inspired by deep
reinforcement learning which enables the evolution of large neural networks.
Although high-performing in many environments, PGA-MAP-Elites fails on several
tasks where the convergent search of the gradient-based variation operator
hinders diversity. In this work, we present three contributions: (1) we enhance
the Policy Gradient variation operator with a descriptor-conditioned critic
that reconciles diversity search with gradient-based methods, (2) we leverage
the actor-critic training to learn a descriptor-conditioned policy at no
additional cost, distilling the knowledge of the population into one single
versatile policy that can execute a diversity of behaviors, (3) we exploit the
descriptor-conditioned actor by injecting it in the population, despite network
architecture differences. Our method, DCG-MAP-Elites, achieves equal or higher
QD score and coverage compared to all baselines on seven challenging continuous
control locomotion tasks.