From Past to Future: Rethinking Eligibility Traces
Dhawal Gupta Scott M. Jordan Shreyas Chaudhari Bo Liu Philip S. Thomas Bruno Castro da Silva
In this paper, we introduce a fresh perspective on the challenges of credit
assignment and policy evaluation. First, we delve into the nuances of
eligibility traces and explore instances where their updates may result in
unexpected credit assignment to preceding states. From this investigation
emerges the concept of a novel value function, which we refer to as the
\emph{bidirectional value function}. Unlike traditional state value functions,
bidirectional value functions account for both future expected returns (rewards
anticipated from the current state onward) and past expected returns
(cumulative rewards from the episode's start to the present). We derive
principled update equations to learn this value function and, through
experimentation, demonstrate its efficacy in enhancing the process of policy
evaluation. In particular, our results indicate that the proposed learning
approach can, in certain challenging contexts, perform policy evaluation more
rapidly than TD($\lambda$) -- a method that learns forward value functions,
$v^\pi$, \emph{directly}. Overall, our findings present a new perspective on
eligibility traces and potential advantages associated with the novel value
function it inspires, especially for policy evaluation.