A Survey of Temporal Credit Assignment in Deep Reinforcement Learning
Eduardo Pignatelli Johan Ferret Matthieu Geist Thomas Mesnard Hado van Hasselt Laura Toni
The Credit Assignment Problem (CAP) refers to the longstanding challenge of
Reinforcement Learning (RL) agents to associate actions with their long-term
consequences. Solving the CAP is a crucial step towards the successful
deployment of RL in the real world since most decision problems provide
feedback that is noisy, delayed, and with little or no information about the
causes. These conditions make it hard to distinguish serendipitous outcomes
from those caused by informed decision-making. However, the mathematical nature
of credit and the CAP remains poorly understood and defined. In this survey, we
review the state of the art of Temporal Credit Assignment (CA) in deep RL. We
propose a unifying formalism for credit that enables equitable comparisons of
state of the art algorithms and improves our understanding of the trade-offs
between the various methods. We cast the CAP as the problem of learning the
influence of an action over an outcome from a finite amount of experience. We
discuss the challenges posed by delayed effects, transpositions, and a lack of
action influence, and analyse how existing methods aim to address them.
Finally, we survey the protocols to evaluate a credit assignment method, and
suggest ways to diagnoses the sources of struggle for different credit
assignment methods. Overall, this survey provides an overview of the field for
new-entry practitioners and researchers, it offers a coherent perspective for
scholars looking to expedite the starting stages of a new study on the CAP, and
it suggests potential directions for future research