Colored Noise in PPO: Improved Exploration and Performance Through
  Correlated Action Sampling
Jakob Hollenstein Georg Martius Justus Piater
Proximal Policy Optimization (PPO), a popular on-policy deep
  reinforcement learning method, employs a stochastic policy for
  exploration. In this paper, we propose a colored noise-based
  stochastic policy variant of PPO. Previous research highlighted the
  importance of temporal correlation in action noise for effective
  exploration in off-policy reinforcement learning. Building on
  this, we investigate whether correlated noise can also enhance
  exploration in on-policy methods like PPO. We discovered that
  correlated noise for action selection improves learning performance
  and outperforms the currently popular uncorrelated white noise
  approach in on-policy methods. Unlike off-policy learning, where pink
  noise was found to be highly effective, we found that a colored
  noise, intermediate between white and pink, performed best for
  on-policy learning in PPO. We examined the impact of varying the
  amount of data collected for each update by modifying the number of
  parallel simulation environments for data collection and observed
  that with a larger number of parallel environments, more strongly
  correlated noise is beneficial. Due to the significant impact and
  ease of implementation, we recommend switching to correlated noise as
  the default noise source in PPO.