From Past to Future: Rethinking Eligibility Traces
This work addresses credit assignment issues in reinforcement learning, offering a novel perspective that could improve policy evaluation, though it appears incremental in nature.
The paper tackles the problem of credit assignment and policy evaluation in reinforcement learning by introducing a bidirectional value function that accounts for both future and past expected returns, showing it can perform policy evaluation more rapidly than TD(λ) in certain challenging contexts.
In this paper, we introduce a fresh perspective on the challenges of credit assignment and policy evaluation. First, we delve into the nuances of eligibility traces and explore instances where their updates may result in unexpected credit assignment to preceding states. From this investigation emerges the concept of a novel value function, which we refer to as the \emph{bidirectional value function}. Unlike traditional state value functions, bidirectional value functions account for both future expected returns (rewards anticipated from the current state onward) and past expected returns (cumulative rewards from the episode's start to the present). We derive principled update equations to learn this value function and, through experimentation, demonstrate its efficacy in enhancing the process of policy evaluation. In particular, our results indicate that the proposed learning approach can, in certain challenging contexts, perform policy evaluation more rapidly than TD($λ$) -- a method that learns forward value functions, $v^π$, \emph{directly}. Overall, our findings present a new perspective on eligibility traces and potential advantages associated with the novel value function it inspires, especially for policy evaluation.