---
abstract: 'In recent years, on-policy reinforcement learning (RL) has been successfully applied to many different continuous control tasks. While RL algorithms are often conceptually simple, their state-of-the-art implementations take numerous low- and high-level design decisions that strongly affect the performance of the resulting agents. Those choices are usually not extensively discussed in the literature, leading to discrepancy between published descriptions of algorithms and their implementations. This makes it hard to attribute progress in RL and slows down overall progress [@implementation_matters]. As a step towards filling that gap, we implement &gt;50 such “choices” in a unified on-policy RL framework, allowing us to investigate their impact in a large-scale empirical study. We train over 250’000 agents in five continuous control environments of different complexity and provide insights and practical recommendations for on-policy training of RL agents.'
author:
- |
    Marcin Andrychowicz, Anton Raichuk, Piotr Stańczyk, Manu Orsini,\
    **Sertan Girgin, Raphael Marinier, Léonard Hussenot, Matthieu Geist,**\
    **Olivier Pietquin, Marcin Michalski, Sylvain Gelly, Olivier Bachem**\
    \
    Google Research, Brain Team
bibliography:
- 'main.bib'
title: 'What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Study'
---

Introduction
============

Deep reinforcement learning (RL) has seen increased interest in recent years due to its ability to have neural-network-based agents learn to act in environments through interactions. For continuous control tasks, on-policy algorithms such as REINFORCE [@pg], TRPO [@trpo], A3C [@a3c], PPO [@ppo] and off-policy algorithms such as DDPG [@ddpg] and SAC [@sac] have enabled successful applications such as quadrupedal locomotion [@sac2], self-driving [@kendall2019learning] or dexterous in-hand manipulation [@sac2; @openai2018learning; @openai2019solving].

Many of these papers investigate in depth different loss functions and learning paradigms. Yet, it is less visible that behind successful experiments in deep RL there are complicated code bases that contain a large number of low- and high-level design decisions that are usually not discussed in research papers. While one may assume that such “choices” do not matter, there is some evidence that they are in fact crucial for or even driving good performance [@implementation_matters].

While there are open-source implementations available that can be used by practitioners, this is still unsatisfactory: In research publications, often different algorithms implemented in different code bases are compared one-to-one. This makes it impossible to assess whether improvements are due to the algorithms or due to their implementations. Furthermore, without an understanding of lower-level choices, it is hard to assess the performance of high-level algorithmic choices as performance may strongly depend on the tuning of hyperparameters and implementation-level details. Overall, this makes it hard to attribute progress in RL and slows down further research [@henderson2018deep; @implementation_matters; @islam2017reproducibility].

**Our contributions.** Our key goal in this paper is to investigate such lower level choices in depth and to understand their impact on final agent performance. Hence, as our key contributions, we (1) implement &gt;50 choices in a unified on-policy algorithm implementation, (2) conducted a large-scale (more than 250’000 agents trained) experimental study that covers different aspects of the training process, and (3) analyze the experimental results to provide practical insights and recommendations for the on-policy training of RL agents.

**Most surprising finding.** While many of our experimental findings confirm common RL practices, some of them are quite surprising, e.g. the policy initialization scheme significantly influences the performance while it is rarely even mentioned in RL publications. In particular, we have found that initializing the network so that the initial action distribution has zero mean, a rather low standard deviation and is independent of the observation significantly improves the training speed (Sec. \[sec:results-arch\]).

The rest of of this paper is structured as follows: We describe our experimental setup and performance metrics used in Sec. \[sec:performance\]. Then, in Sec. \[sec:results\] we present and analyse the experimental results and finish with related work in Sec. \[sec:related\] and conclusions in Sec. \[sec:conclusions\]. The appendices contain the detailed description of all design choices we experiment with (App. \[sec:choices\]), default hyperparameters (App. \[sec:default-settings\]) and the raw experimental results (App. \[exp\_final\_losses\] - \[exp\_final\_regularizer\]).

Study design {#sec:performance}
============

#### Considered setting.

In this paper, we consider the setting of *on-policy reinforcement learning for continuous control*. We define on-policy learning in the following loose sense: We consider policy iteration algorithms that iterate between generating experience using the current policy and using the experience to improve the policy. This is the standard *modus operandi* of algorithms usually considered on-policy such as PPO [@ppo]. However, we note that algorithms often perform several model updates and thus may operate technically on off-policy data within a single policy improvement iteration. As benchmark environments, we consider five widely used continuous control environments from OpenAI Gym [@gym] of varying complexity: Hopper-v1, Walker2d-v1, HalfCheetah-v1, Ant-v1, and Humanoid-v1 [^1].

#### Unified on-policy learning algorithm.

We took the following approach to create a highly configurable unified on-policy learning algorithm with as many choices as possible:

1.  We researched prior work and popular code bases to make a list of commonly used choices, i.e., different loss functions (both for value functions and policies), architectural choices such as initialization methods, heuristic tricks such as gradient clipping and all their corresponding hyperparameters.

2.  Based on this, we implemented a single, unified on-policy agent and corresponding training protocol starting from the SEED RL code base [@seed]. Whenever we were faced with implementation decisions that required us to take decisions that could not be clearly motivated or had alternative solutions, we further added such decisions as additional choices.

3.  We verified that when all choices are selected as in the PPO implementation from OpenAI baselines, we obtain similar performance as reported in the PPO paper [@ppo]. We chose PPO because it is probably the most commonly used on-policy RL algorithm at the moment.

The resulting agent implementation is detailed in Appendix \[sec:choices\]. The key property is that the implementation exposes all choices as configuration options in an unified manner. For convenience, we mark each of the choice in this paper with a number (e.g., ) and a fixed name (e.g. ) that can be easily used to find a description of the choice in Appendix \[sec:choices\].

#### Difficulty of investigating choices.

The primary goal of this paper is to understand how the different choices affect the final performance of an agent and to derive recommendations for these choices. There are two key reasons why this is challenging:

First, we are mainly interested in insights on choices for good hyperparameter configurations. Yet, if all choices are sampled randomly, the performance is very bad and little (if any) training progress is made. This may be explained by the presence of sub-optimal settings (e.g., hyperparameters of the wrong scale) that prohibit learning at all. If there are many choices, the probability of such failure increases exponentially.

Second, many choices may have strong interactions with other related choices, for example the learning rate and the minibatch size. This means that such choices need to be tuned together and experiments where only a single choice is varied but interacting choices are kept fixed may be misleading.

#### Basic experimental design.

To address these issues, we design a series of experiments as follows: We create groups of choices around thematic groups where we suspect interactions between different choices, for example we group together all choices related to neural network architecture. We also include in all of the groups as we suspect that it may interact with many other choices.

Then, in each experiment, we train a large number of models where we randomly sample the choices within the corresponding group [^2]. All other settings (for choices not in the group) are set to settings of a competitive base configuration (detailed in Appendix \[sec:default-settings\]) that is close to the default PPOv2 configuration[^3] scaled up to $256$ parallel environments. This has two effects: First, it ensures that our set of trained models contains good models (as verified by performance statistics in the corresponding results). Second, it guarantees that we have models that have different combinations of potentially interacting choices.

We then consider two different analyses for each choice (e.g, for ):

*Conditional 95th percentile*: For each potential value of that choice (e.g., = `N-Step`), we look at the performance distribution of sampled configurations with that value. We report the 95th percentile of the performance as well as a confidence interval based on a binomial approximation [^4]. Intuitively, this corresponds to a robust estimate of the performance one can expect if all other choices in the group were tuned with random search and a limited budget of roughly 20 hyperparameter configurations.

*Distribution of choice within top 5% configurations.* We further consider for each choice the distribution of values among the top 5% configurations trained in that experiment. The reasoning is as follows: By design of the experiment, values for each choice are distributed uniformly at random. Thus, if certain values are over-represented in the top models, this indicates that the specific choice is important in guaranteeing good performance.

#### Performance measures.

We employ the following way to compute performance: For each hyperparameter configuration, we train $3$ models with independent random seeds where each model is trained for one million (Hopper, HalfCheetah, Walker2d) or two million environment steps (Ant, Humanoid). We evaluate trained policies every hundred thousand steps by freezing the policy and computing the average undiscounted episode return of 100 episodes (with the stochastic policy). We then average these score to obtain a single performance score of the seed which is proportional to the area under the learning curve. This ensures we assign higher scores to agents that learn quickly. The performance score of a hyperparameter configuration is finally set to the median performance score across the 3 seeds. This reduces the impact of training noise, i.e., that certain seeds of the same configuration may train much better than others.

Related Work {#sec:related}
============

Islam et al. [@islam2017reproducibility] and Henderson et al. [@henderson2018deep] point out the reproducibility issues in RL including the performance differences between different code bases, the importance of hyperparameter tuning and the high level of stochasticity due to random seeds. Tucker et al. [@tucker2018mirage] showed that the gains, which had been attributed to one of the recently proposed policy gradients improvements, were, in fact, caused by the implementation details. The most closely related work to ours is probably Engstrom et al. [@implementation_matters] where the authors investigate code-level improvements in the PPO [@ppo] code base and conclude that they are responsible for the most of the performance difference between PPO and TRPO [@trpo]. Our work is also similar to other large-scale studies done in other fields of Deep Learning, e.g. model-based RL [@langlois2019benchmarking], GANs [@lucic2018gans], NLP [@kaplan2020scaling], disentangled representations [@locatello2018challenging] and convolution network architectures [@radosavovic2020designing].

Conclusions {#sec:conclusions}
===========

In this paper, we investigated the importance of a broad set of high- and low-level choices that need to be made when designing and implementing on-policy learning algorithms. Based on more than 250’000 experiments in five continuous control environments, we evaluate the impact of different choices and provide practical recommendations. One of the surprising insights is that the initial action distribution plays an important role in agent performance. We expect this to be a fruitful avenue for future research.

[^1]: It has been noticed that the version of the Mujoco physics simulator [@mujoco] can slightly influence the behaviour of some of the environments — <https://github.com/openai/gym/issues/1541>. We used Mujoco 2.0 in our experiments.

[^2]: Exact details for the different experiments are provided in Appendices \[exp\_final\_losses\] - \[exp\_final\_regularizer\].

[^3]: <https://github.com/openai/baselines/blob/master/baselines/ppo2/defaults.py>

[^4]: We compute confidence intervals with a significance level of $\alpha=5\%$ as follows: We find $i_l = icdf\left(\frac\alpha2\right)$ and $i_h = icdf\left(1-\frac\alpha2\right)$ where $icdf$ is the inverse cumulative density function of a binomial distribution with $p=0.95$ (as we consider the 95th percentile) and the number of draws equals the number of samples. We then report the $i_l$th and $i_h$th highest scores as the confidence interval.
