Reward Scale Robustness for Proximal Policy Optimization via DreamerV3
  Tricks
Ryan Sullivan Akarsh Kumar Shengyi Huang John P. Dickerson Joseph Suarez
Most reinforcement learning methods rely heavily on dense, well-normalized
environment rewards. DreamerV3 recently introduced a model-based method with a
number of tricks that mitigate these limitations, achieving state-of-the-art on
a wide range of benchmarks with a single set of hyperparameters. This result
sparked discussion about the generality of the tricks, since they appear to be
applicable to other reinforcement learning algorithms. Our work applies
DreamerV3's tricks to PPO and is the first such empirical study outside of the
original work. Surprisingly, we find that the tricks presented do not transfer
as general improvements to PPO. We use a high quality PPO reference
implementation and present extensive ablation studies totaling over 10,000 A100
hours on the Arcade Learning Environment and the DeepMind Control Suite. Though
our experiments demonstrate that these tricks do not generally outperform PPO,
we identify cases where they succeed and offer insight into the relationship
between the implementation tricks. In particular, PPO with these tricks
performs comparably to PPO on Atari games with reward clipping and
significantly outperforms PPO without reward clipping.