Skill or Luck? Return Decomposition via Advantage Functions
Hsiao-Ru Pan Bernhard Schölkopf
Learning from off-policy data is essential for sample-efficient reinforcement
learning. In the present work, we build on the insight that the advantage
function can be understood as the causal effect of an action on the return, and
show that this allows us to decompose the return of a trajectory into parts
caused by the agent's actions (skill) and parts outside of the agent's control
(luck). Furthermore, this decomposition enables us to naturally extend Direct
Advantage Estimation (DAE) to off-policy settings (Off-policy DAE). The
resulting method can learn from off-policy trajectories without relying on
importance sampling techniques or truncating off-policy actions. We draw
connections between Off-policy DAE and previous methods to demonstrate how it
can speed up learning and when the proposed off-policy corrections are
important. Finally, we use the MinAtar environments to illustrate how ignoring
off-policy corrections can lead to suboptimal policy optimization performance.