---
abstract: 'We propose a novel reinforcement learning algorithm, , that incorporates the strengths of off-policy RL algorithms into Quality Diversity (QD) approaches. Quality-Diversity methods contribute structural biases by decoupling the search for diversity from the search for high return, resulting in efficient management of the exploration-exploitation trade-off. However, these approaches generally suffer from sample inefficiency as they call upon evolutionary techniques. removes this limitation by relying on off-policy RL algorithms. More precisely, we train a population of off-policy deep RL agents to simultaneously maximize diversity inside the population and the return of the agents. selects agents from the diversity-return Pareto Front, resulting in stable and efficient population updates. Our experiments on the environment show that can solve challenging exploration and control problems with deceptive rewards while being more than 15 times more sample efficient than its evolutionary counterparts.'
author:
- |
    Geoffrey Cideron\
    InstaDeep\
    `g.cideron@instadeep.com`\
    Thomas Pierrot\
    InstaDeep\
    `t.pierrot@instadeep.com`\
    Nicolas Perrin\
    CNRS, Sorbonne Université\
    `perrin@isir.upmc.fr`\
    Karim Beguir\
    InstaDeep\
    `kb@instadeep.com`\
    Olivier Sigaud\
    Sorbonne Université\
    `olivier.sigaud@upmc.fr`\
bibliography:
- 'ms.bib'
title: 'QD-RL: Efficient Mixing of Quality and Diversity in Reinforcement Learning'
---

Introduction
============

Despite outstanding successes in specific domains such as games [@silver2017mastering; @jaderberg2019human] and robotics [@tobin2018domain; @akkaya2019solving], Reinforcement Learning (RL) algorithms are still far from being immediately applicable to complex sequential decision problems. Among the issues, a remaining burden is the need to find the right balance between exploitation and exploration. On one hand, algorithms which do not explore enough can easily get stuck in poor local optima. On the other hand, exploring too much hinders sample efficiency and can even prevent users from applying RL to large real world problems.

Dealing with this exploration-exploitation trade-off has been the focus of many RL papers [@tang2016exploration; @bellemare2016unifying; @fortunato2017noisy; @plappert2017parameter]. Among other things, having a population of agents working in parallel in the same environment is now a common recipe to stabilize learning and improve exploration, as these parallel agents collect a more diverse set of samples. This has led to two approaches, namely [*distributed RL*]{} where the agents are the same and [*population-based training*]{}, where diversity between agents further favors exploration [@jung2020population; @parker2020effective]. However, such methods do certainly not make the most efficient use of available computational resources, as the agents may collect highly redundant information. Besides, the focus on sparse or deceptive rewards problems led to the realization that looking for diversity independently from maximizing rewards might be a good exploration strategy [@lehman2011abandoning; @eysenbach2018diversity; @colas2018gep]. More recently, it was established that if one can define a [*behavior space*]{} or [*outcome space*]{} corresponding to the smaller space that matters to decide if a behavior is successful or not, maximizing diversity in this space might be the optimal strategy to find the sparse reward source [@doncieux2019novelty].

When the reward signal is not sparse though, one can do better than just looking for diversity. Trying to simultaneously maximize diversity and rewards has been formalized into the Quality-Diversity (QD) framework [@pugh2016quality; @cully2017quality]. The corresponding algorithms try to populate the outcome space as widely as possible with an [*archive*]{} of past solutions which are both diverse and reward efficient. To do so, they generally rely on evolutionary algorithms. Selecting diverse and reward efficient solutions is then performed using the Pareto front of the *diversity* $\times$ *reward efficiency* landscape, or populating a grid of outcome cells with reward efficient solutions in the algorithm [@mouret2015illuminating]. In principle, the QD approach offers a great way to deal with the exploration-exploitation trade-off as it simultaneously ensures pressure towards both wide covering of the outcome space and high return efficiency. However, these methods suffer from relying on evolutionary methods. Though they have been shown to be competitive with deep RL approaches provided enough computational power [@salimans2017evolution; @colas2020scaling], they do not take advantage of the gradient’s analytical form, and thus have to sample to estimate gradients, resulting in far worse sample efficiency than their deep RL counterparts [@sigaud2019policy].

On the other hand, deep RL methods which leverage policy gradients have far better sample efficiency but they struggle on problems that require strong exploration and are sensitive to poorly conditioned reward signals such as deceptive rewards [@colas2018gep]. This is in part because they explore in the action space, the state-action space or the policy space rather than in an outcome space.

In this work, we combine the general QD framework with policy gradient methods and capitalize on the strengths of both approaches. Our algorithm explores in an outcome space and thus can solve problems that simultaneously require complex exploration and high dimensional control capabilities. We investigate the properties of by first controlling a low dimensional agent in a maze, and then addressing , a larger benchmark. We compare to several recent algorithms which also combine a diversity objective and a return maximization method, namely the family which mixes evolution strategies with novelty search [@conti2017improving] and the algorithm [@colas2020scaling] which uses to maintain a diverse and high performing population. The latter has been shown to scale well enough to also address large benchmarks, but we show that is several orders of magnitude more sample efficient than these competitors.

Related Work {#sec:related}
============

We consider the general context of a fully observable Markov Decision Problem (MDP) $\left( \mathcal{S}, \mathcal{A}, \mathcal{T}, \mathcal{R}, \gamma \right)$ where $\mathcal{S}$ is the state space, $\mathcal{A}$ is the action space, $\mathcal{T}: \mathcal{S} \times \mathcal{A} \rightarrow \mathcal{S}$ is the transition function, $\mathcal{R}: \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R}$ is the reward function and $\gamma$ is a discount factor. The exploration-exploitation trade-off being central in RL, the search for efficient exploration methods are ubiquitous in the domain. We focus on the relationship between our work and two methods: those which introduce explicit diversity into a multi-actor deep RL approach, and those which combine distinct mechanisms for exploration and exploitation.

#### Diversity in multi-actor RL

Managing several actors is now a well established method to improve wall clock time and stabilize learning [@jaderberg2017population]. But including an explicit diversity criterion is a more recent trend.

The algorithm [@doan2019attraction] uses a combination of attraction and repulsion mechanisms between good agents and poor agents to ensure diversity in a population of agents trained in parallel. The algorithm shows improvement in performance in large continuous action benchmarks such as and sparse reward variants. But diversity is defined in the space of policy performance thus the drive towards novel behaviors could be strengthened.

The algorithm [@jung2020population] is an instance of population-based training where the parameters of the best actor are softly distilled into the rest of the population. To prevent the whole population from collapsing into a single agent, a simple diversity criterion is enforced so as to maintain a minimum distance between all agents. The algorithm shows good performance over a large set of continuous action benchmarks, including “delayed” variants where the reward is obtained only every $K$ time steps. However, the diversity criterion they use is far from guaranteeing efficient exploration of the outcome space, particularly in the absence of reward, and it seems that the algorithms mostly benefits from the higher stability of population-based training.

With respect to , the algorithm [@parker2020effective] proposes a population-wide diversity criterion which consists in maximizing the volume between the parameters of the agents in a latent space. This criterion better limits redundancy between the considered agents.

Like our work, all these methods use a population of deep RL agents and explicitly look for diversity among these agents. However, none of them addresses deceptive reward environments such as the mazes we consider in our work. Furthermore, none of them clearly separates two components nor searches for diversity in the outcome space as does.

#### Separated exploration and exploitation mechanisms

One extreme case of the separation between exploration and exploitation is “exploration-only” methods. The efficiency of this approach was first put forward within the evolutionary optimization literature [@lehman2011abandoning; @doncieux2019novelty] and then imported into the reinforcement learning literature with works such as [@eysenbach2018diversity] which gave rise to several recent follow-up [@pong2019skew; @lee2019efficient; @islam2019marginalized]. These methods have proven useful in the sparse reward case, but they are inherently limited when some reward signal can be used and maximized during exploration. A second approach is sequential combination. Similarly to us, the algorithm [@colas2018gep] combines a diversity seeking component, namely [*Goal Exploration Processes*]{} [@forestier2017intrinsically] and a deep RL algorithm, namely [@lillicrap2015continuous] and shows that combining them sequentially can overcome a deceptive gradient issue. This sequential combination of exploration-then-exploitation is also present in [@ecoffet2019go] which explores first and then memorizes the sequence to look for a high reward policy in games, and in [@matheron2020pbcs] which does the same in a continuous action domain. Again, this approach is limited when the reward signal can help driving the exploration process towards a satisfactory solution.

Removing the sequentiality limitation, some approaches use a population of agents with various exploration rates [@badia2020agent57]. Along a different line, the algorithm [@pourchot2018cem] combines an evolutionary algorithm, [@deboer05tutorial], and a deep RL algorithm, [@fujimoto2018addressing] in such a way that each component takes the lead when it is the most appropriate in the current situation. Doing so, benefits from the better sample efficiency of deep RL and from the higher stability of evolutionary methods. But the evolutionary part is not truly a diversity seeking component and, being still an evolutionary method, it is not as sample efficient as . A common feature between and our work is that the reward seeking agents benefit from the findings of the other agents by sharing their replay buffer.

Closer to our quality-diversity inspired approach, [@conti2017improving] proposes and . But, as outlined in [@colas2020scaling], these approaches are not sample efficient and the diversity and environment reward functions are mixed in a less efficient way. The most closely related work w.r.t. ours is [@colas2020scaling]. The algorithm also optimizes both diversity and reward efficiency, using an archive and two ES populations. Instead of using a Pareto front, uses the approach where the outcome space is split in cells that the algorithm has to cover. Using such distributional ES approach has been shown to be critically more efficient than population-based GA methods [@salimans2017evolution], but our results show that they are still less sample efficient than off-policy deep RL methods as they do not leverage direct access to the policy gradient.

QD-RL
=====

We present , a quality-diversity optimization method designed to address hard exploration problems where sample efficiency matters. As depicted in  \[fig:gen\_schema\], optimizes a population of agents for both environment reward and diversity using off-policy policy gradient methods which are known to be more efficient than traditional genetic algorithms or evolution strategies [@salimans2017evolution; @such2017deep]. In this study, we chose to rely on the agent (see Supplementary Section \[sec:td3\]) but any other off-policy agent such as [@haarnoja2018soft] could be used instead.

With respect to the standard MDP framework, introduces an extra outcome space $O$ and a behavior characterization function $\bfunc: \mathcal{S} \rightarrow \mathcal{O}$ that extracts the outcome $o$ for a state $s$. The outcome of a behavior characterizes what matters about this behavior. As it often corresponds to what is needed to determine whether the behavior was successful or not, this outcome space can be equivalent to a goal space such as introduced in [@schaul2015universal; @andrychowicz2017hindsight]. For example, when working in a maze environment, the outcome may represent the coordinates $(x, y)$ at the end of the trajectory of the agent, which may also be its goal. However, in contrast to , we do not condition the policy on the outcome $o$. In this work, the behavior characterization function is given, as is also the case in [@schaul2015universal; @andrychowicz2017hindsight], and we consider outcomes computed as a function of a single state. The more general case where it is learned or computed as a function of the whole trajectory is left for future work.

As any QD system, manages a population of policies into an archive which contains all previously trained actors. The first generation contains a population of $N$ neural actors $\pi_{\theta_i}: \mathcal{S} \rightarrow \mathcal{A}$ with parameters $\theta_i, i \in [1, N ]$. While all these actors share the same neural architecture, their weights are initialized differently. At each iteration, a selection mechanism based on the Pareto front selects $N$ best actors from the archive containing all past agents, according to two criteria: the environment reward and a measure of novelty of the actor. To better stick to the QD framework, hereafter the former is called “quality” and the latter “diversity”. If the Pareto front contains less than $N$ actors, the whole Pareto front is selected and computed again over the remaining actors. Additional actors are sampled from the new Pareto front, and so on until $N$ actors are sampled.

These selected actors form a new generation. Then, half of the selected actors are trained to optimize quality while the others optimize diversity. These actors with updated weights are then evaluated in the environment and added to the archive. In more details, training is performed as follows.

![Architecture of the algorithm. []{data-label="fig:gen_schema"}](images/QDRL-3.png){width="89.00000%"}

First, as any standard RL method, optimizes actors so as to maximize quality. More formally, it updates the actor weights to maximize the objective function $J_{\text{quality}}(\theta_i) = \Esp_{\tau_i}{\left[ \sum_t \gamma^t r_t^{Q} \right] }$ where $\tau_i$ is a trajectory obtained by following the policy $\pi_{\theta_i}$ and $r_t^{Q}=r$ is the environment reward function.

Second, also optimizes actors to increase diversity in the outcome space. To evaluate the diversity of an outcome $o$, we seek for the $k$-nearest neighbors of outcome $o$ in the archive and compute a novelty score $NS(o, A)$ as the mean of the squared Euclidean distances between $o$ and its $k$ neighbors, as in [@lehman2011abandoning; @conti2017improving]. More formally, maximizes the objective function $J_{\text{diversity}}(\theta_i) = \Esp_{\tau_i} \left[ \sum_t \gamma^t NS(o_t,  A) \right]$ where $o_t = \bfunc(s_t)$ is the outcome discovered by policy $i$ at time step $t$, $NS()$ is the novelty score function and $A$ is the archive containing already discovered outcomes.

The $J_{\text{quality}}$ and $J_{\text{diversity}}$ functions have the same structure as we can re-write $J_{\text{diversity}}(\theta_i) =  \Esp_{\tau_i}{\left[ \sum_t \gamma^t r^D_t \right] }$, where $r^D = NS(o, A)$ is a non stationary reward function corresponding to novelty scores. Thus all the mechanisms introduced in the deep RL literature to optimize $J_{\text{quality}}$ can also be applied to optimize $J_{\text{diversity}}$. Notably, we can introduce Q-value functions $Q_Q$ and $Q_D$ dealing with quality and diversity and we can define two randomly initialized critic neural networks $Q_{\theta^Q}$ and $Q_{\theta^D}$, with parameters $\theta^Q$ and $\theta^D$ to approximate these functions. These critics are shared by all the trained actors. Therefore, they capture the average population performance rather than the performance of individual actors, which has both an information sharing effect and a smoothing effect. We found that training individual critics is harder in practice and left this analysis for future work.

The quality and diversity update of actor weights is performed according to Equation . An update consists in sampling a batch of transitions from the replay buffer and optimizing the weights of both critics so as to maximize quality and diversity. Then, we optimize parameters of half policies so as to maximize $J_{\text{quality}}$ and the other half to maximize $J_{\text{diversity}}$. Therefore, the global update can be written

$$\label{eq:global_Q_update}
        \left\{
            \begin{array}{ll}
              \theta^Q \leftarrow \theta^Q - \frac{2\alpha}{N} \nabla_{\theta^Q} \sum\limits_{batch} \sum\limits_{i=1}^{\nicefrac{N}{2}} \left( Q_{\theta^Q}(s_t, a_t) - (r^Q_t + \gamma Q_{\theta^{Q\prime}}(s_{t+1}, \pi_{\theta_i}(s_{t+1})))  \right)^2 \\
              
              \theta^D \leftarrow \theta^D - \frac{2\alpha}{N} \nabla_{\theta^D} \sum\limits_{batch} \sum\limits_{i=\nicefrac{N}{2} + 1}^{N} \left( Q_{\theta^D}(s_t, a_t) - (r^D_t + \gamma Q_{\theta^{D\prime}}(s_{t+1}, \pi_{\theta_i}(s_{t+1})))  \right)^2\\
            \end{array}
          \right.$$

$$\label{eq:global_pi_update}
        \left\{
            \begin{array}{ll}
              
              \theta_{i} \leftarrow \theta_{i} + \alpha \nabla_{\theta_i} \sum\limits_{batch} Q_{\theta^Q} (s_t, \pi_{\theta_i}(s_t)), \ \ \forall i \leq \nicefrac{N}{2} \hspace{0.2cm} \textit{// Quality update of half actors}\\
              
              \theta_{i} \leftarrow \theta_{i} + \alpha \nabla_{\theta_i} \sum\limits_{batch} Q_{\theta^D} (s_t, \pi_{\theta_i}(s_t)), \ \ \forall i > \nicefrac{N}{2} \hspace{0.2cm} \textit{// Diversity update of half actors}
            \end{array}
          \right.$$

where ${\theta^{Q}}^\prime$ and ${\theta^{D}}^\prime$ correspond to the parameters of target critic networks. To keep notations simple, updates of the extra critic networks introduced in to reduce the value estimation bias do not appear in , but we use them in practice.

Once updates have been performed, trajectories are collected in parallel from all policies. These trajectories are stored into a common replay buffer and the tuple (final outcome $o_{T}$, return, parameters) is stored into the archive. Since the novelty score of an outcome varies through time as the archive grows, instead of storing it, we store outcomes and fresh diversities are computed every time a batch is sampled from the replay buffer.

A short version of the algorithm is presented in Algorithm \[alg:short\]. Other implementation details are presented in Supplementary Section \[sec:details\].

Experiments
===========

In this section, we demonstrate the capability of to solve challenging exploration problems. We implement it with the algorithm and refer to this implementation as the algorithm. Hyper-parameters are described in Section \[sec:details\] of the supplementary document. We first analyse each component of and demonstrate their usefulness on a toy example. Then we show that can solve a more challenging control and exploration problem such as navigating the Ant into a large maze with a better sample complexity than its standard evolutionary competitors.

[.5]{} ![Experimental environments. Though they may look similar, the environment state and action spaces are two-dimensional, whereas in , the state space has 29 dimensions and the action space 8.[]{data-label="fig:envs"}](images/al-toy_maze_vierge.png "fig:"){width="0.9\linewidth"}

[.5]{} ![Experimental environments. Though they may look similar, the environment state and action spaces are two-dimensional, whereas in , the state space has 29 dimensions and the action space 8.[]{data-label="fig:envs"}](images/antobstacle.png "fig:"){width="0.9\linewidth"}

Point Maze: Move a point in a Maze
----------------------------------

We first consider the environment in which the agent controls a 2D material point which must exit from a three corridors maze depicted in  \[fig:toy\_maze\]. The observation corresponds to the agent coordinates $(x_t, y_t) \in [-1,1]^2$ at time $t$. The two continuous actions $(\delta x, \delta y) \in [-0.1,0.1]^2$ correspond to position increments along the $x$ and $y$ axes. The outcome space is the final state $(x_f,y_f)$ of the agent, as in [@conti2017improving]. The initial position of the agent is sampled uniformly in $[-0.1, 0.1]\times[-1, -0.7]$. This zone is located at the bottom right of the maze. The exit area is a square centered at $(x_{\text{goal}}=-0.5, y_{\text{goal}}=0.8)$ of width $0.1$. Once this exit square is reached, the episode ends. The maximum length of an episode is 200 time steps. The reward is computed as $r_t = - (x_t - x_{\text{goal}})^2 - (y_t - y_{\text{goal}})^2$. This reward leads to a deceptive gradient signal: following it would lead the agent to stay stuck by the second wall in the maze, as shown in  \[fig:grad\_maps\] of Supplementary Section \[sec:analysis\_supp\]. In order to exit the maze, the agent must find the right balance between exploitation and exploration, that is at a certain point ignore the policy gradient and only explore the maze. Thus, though this example may look simple due to its low dimension, it remains very challenging for standard deep reinforcement agents such as .

performs three main operations: (i) it optimizes half of the agents to maximize quality; (ii) it does the same to the other half to maximize diversity; (iii) it uses a quality-diversity Pareto front as a population selection mechanism. We investigate the impact of each of these components separately through an ablation study. For all experiments, we use 4 actors. Results are aggregated in  \[fig:aggcirclezone\].

First, we measure performance when training the 4 actors to maximize quality only. We call the resulting agent , but this is simply a multi-actor . As depicted in  \[fig:coverage\_map\_pt\_maze\] of Supplementary Section \[sec:analysis\_supp\], the population finds a way to the second maze wall but is stuck there due to the deceptive nature of the gradient. This experiment shows clearly enough that using a quality-only strategy has no chance of solving hard exploration problems with a deceptive reward signal such as or .

Then, we evaluate the agent performance when training the 4 actors to maximize diversity only. We call the resulting agent . We show that finds sometimes how to get further the second wall but with a large variance and without finding the optimal trajectory.

We then consider a + agent that optimizes only for diversity but performs agent selection from the archive with a Pareto front, that is it selects 4 actors for the next generation based on both their quality and diversity, but without optimizing the former. Interestingly, adding the Pareto front selection mechanism significantly improves performance and stability.

Finally, optimizes half of the actors for quality, the other half for diversity and it selects them with the Pareto Front mechanism. We observe that outperforms all ablations, even if the improvement over + is lesser, which means that optimizing for quality is less critical in this environment as good enough solutions are found just by maximising diversity.  \[tab:recap\] summarises all the ablations we performed.

[.5]{} ![Learning curves of versus ablations on and several other agents on . The performance corresponds to the highest return on and to the highest negative distance to the goal area on .[]{data-label="fig:results"}](images/agg_clean_circle_maze.pdf "fig:"){width="95.00000%"}

[.5]{} ![Learning curves of versus ablations on and several other agents on . The performance corresponds to the highest return on and to the highest negative distance to the goal area on .[]{data-label="fig:results"}](images/agg_ant_maze_new_colors.pdf "fig:"){width="95.00000%"}

       Opt. Quality   Opt. Diversity   Pareto selection   Episode return ($\pm$ std)
  --- -------------- ---------------- ------------------ ----------------------------
                                                               $-29 \: (\pm 1)$
  +         X                                                  $-35 \:(\pm 3)$
            X                                 X               $-111 \:(\pm 59)$
  +                         X                                  $-128 \:(\pm 0)$
                            X                 X                $-128 \:(\pm 0)$

  : Summary of compared ablations.\[tab:recap\]

Ant Maze: Control an articulated Ant to solve a Maze
----------------------------------------------------

We then test on a challenging environment modified from OpenAI Gym [@brockman2016openai] based on and also used in [@colas2020scaling; @frans2017meta]. We refer to this environment as the environment. In , a four-legged “ant” robot has to reach a goal zone located at $[35, -25]$ which corresponds to the lower right part of the maze (colored in green in ). The initial position of the ant is sampled from a small circle of radius $0.1$ around the initial point $[-25, -25]$ situated in the extreme bottom right of the maze. Maze walls are organized so that following the gradient of distance to the goal drives the ant into a dead-end. As in the , the reward is expressed as minus the distance between the center of gravity of the ant and the center of the goal zone, thus leading to a strongly deceptive gradient. This environment is more complex than as the agent must learn to control a body with 8 degrees of freedom in all directions to explore the maze and solve it. Therefore, this problem is much harder than the standard gym environment in which the ant only learns to go straight forward. The observation space contains the positions, angle, velocities and angular velocities of most ant articulations and center of gravity, and has 29 dimensions. The action space is $[-1,1]^8$ where an action correspond to the choice of 8 continuous torque intensities to apply to the 8 ant articulations. The episodes have a fixed length of 3000 time steps. As previously, the outcome is computed as the final position $(x_f,y_f)$ of the center of gravity of the ant.

We compare the performance of on this benchmark to 4 state-of-the art methods: , , and . While and -optimize only for diversity, and -optimizes only for quality, , and -- optimize for both. To ensure fair comparison, we did not implement our own versions of these algorithms but reused results from the paper [@colas2020scaling]. We also ensured that the environment we used was rigorously the same.

All the baselines were run for 5 seeds. In , each seed corresponds to 20 actors distributed on 20 cores. As in [@colas2020scaling], we compute the average and standard deviation between the seeds of the minimum distance to the goal reached at the end of an episode by one of the agents and report the results in  \[fig:qdrlantmaze\]. As explained in [@colas2020scaling], and -obtain a score around -26, which means that they get stuck in the dead-end, similarly to . By contrast, all other algorithms manage to avoid it. More importantly, the algorithm achieves a similar score to these better exploring agents, but in more than **15 times less samples** than its evolutionary competitors, as shown in Table \[tab:recapant\].

            **Final Perf.** ($\pm$ std)   **Sampled steps**   **Steps to -10**   **Ratio to QD-TD3**
  -------- ----------------------------- ------------------- ------------------ ---------------------
  (Ours)        $-10 \: (\pm 10.1)$              6e8                5e8                   1
                 $-9 \: (\pm 5.6)$              1e10                8e9                  16
                 $-27 \: (\pm 0)$               1e10              $\infty$            $\infty$
                 $-10 \: (\pm 11)$              1e10                8e9                  16
  -              $-10 \: (\pm 6)$               1e10                9e9                  18
  -            $-11.7 \: (\pm 6.9)$             1e10                9e9                  18
  -             $-25.8 \: (\pm 0)$              1e10              $\infty$            $\infty$
                 $-26 \: (\pm 2)$                3e8              $\infty$            $\infty$

  : Summary of compared algorithms. The mean performance is computed as the average over $5$ seeds of the minimum distance to the goal reached at the end of an episode by the population. “Steps to -10” correspond to the number of steps to reach an average performance of -10. - [ee]{} stands for - -.\[tab:recapant\]

These results show that leveraged the sample efficiency brought by off-policy policy gradient to learn to efficiently explore the maze. We also emphasize the low cost of compared to its evolutionary counterparts. To solve the , [**requires only 2 days of training on 20 cores**]{} with no while evolutionary algorithms are usually run on much larger infrastructures. For instance, needs to sample 10.000 different set of parameters per iteration and evaluates them all to compute a diversity gradient with [@colas2020scaling]. Besides, the failure of the ablation into unsurprisingly shows that a pure RL approach without a diversity component fails in these deceptive gradient benchmarks.

Conclusion
==========

In this paper, we proposed a novel way to deal with the exploration-exploitation trade-off by combining a reward-seeking component, a diversity-seeking component and a selection component inspired from the Quality-Diversity approach. Crucially, we showed that quality and diversity could be optimized with off-policy reinforcement learning algorithms, resulting in a significantly improved sample efficiency. We showed experimentally the effectiveness of the resulting framework, which can solve in two days with 20 problems which were previously out of reach without a much larger infrastructure.

Key components of are selection through a Pareto front and the search for diversity in an outcome space. Admittedly, the outcome space needed to compute the diversity reward is hard coded. There are attempts to automatically obtain the outcome space through unsupervised learning methods [@pere2018unsupervised; @paolo2019unsupervised], but defining such a space is often a trivial decision which helps a lot, and can alleviate the need to carefully design reward functions.

In the future, we first want to address the case where the outcome depends on the whole trajectory. Next we plan to further study the versatility of our approach to exploration compared to other deep reinforcement learning exploration approaches. Besides, we intend to show that our approach could be extended to problems where the environment reward function can itself be decomposed into several loosely dependent components, such as standing, moving forward and manipulating objects for a humanoid or solving multiagent reinforcement learning problems. In such environments, we could replace the maximization of the sum of reward contributions with a multi-criteria selection from a Pareto front where diversity would be only one of the considered criteria.

Broader Impact {#broader-impact .unnumbered}
==============

Our paper presents a novel approach to the combination of diversity-driven exploration and modern reinforcement learning techniques. It results in more stable learning with respect to standard reinforcement learning, and more sample efficient learning with respect to standard evolutionary approaches to diversity. We believe this has a positive impact in making reinforcement learning techniques more accessible and feasible towards real world applications. Besides, our work may help casting a needed bridge between the reinforcement learning and the evolutionary optimization research communities. Finally, by releasing our code, we believe that we help efforts in reproducible science and allow the wider community to build upon and extend our work in the future.
