Dissecting Deep RL with High Update Ratios: Combatting Value
  Overestimation and Divergence
Marcel Hussing Claas Voelcker Igor Gilitschenski Amir-massoud Farahmand Eric Eaton
We show that deep reinforcement learning can maintain its ability to learn
without resetting network parameters in settings where the number of gradient
updates greatly exceeds the number of environment samples. Under such large
update-to-data ratios, a recent study by Nikishin et al. (2022) suggested the
emergence of a primacy bias, in which agents overfit early interactions and
downplay later experience, impairing their ability to learn. In this work, we
dissect the phenomena underlying the primacy bias. We inspect the early stages
of training that ought to cause the failure to learn and find that a
fundamental challenge is a long-standing acquaintance: value overestimation.
Overinflated Q-values are found not only on out-of-distribution but also
in-distribution data and can be traced to unseen action prediction propelled by
optimizer momentum. We employ a simple unit-ball normalization that enables
learning under large update ratios, show its efficacy on the widely used
dm_control suite, and obtain strong performance on the challenging dog tasks,
competitive with model-based approaches. Our results question, in parts, the
prior explanation for sub-optimal learning due to overfitting on early data.