Source Traces for Temporal Difference Learning
This work addresses a bottleneck in reinforcement learning for agents needing efficient generalization, though it is incremental as it builds on successor representations and model-based methods.
The paper tackles the problem of slow generalization in temporal difference learning by introducing source traces, which propagate errors to potential causal states, leading to faster learning. It proposes a TD(λ)-like algorithm with proven convergence and a novel method for learning the source map that outperforms previous approaches.
This paper motivates and develops source traces for temporal difference (TD) learning in the tabular setting. Source traces are like eligibility traces, but model potential histories rather than immediate ones. This allows TD errors to be propagated to potential causal states and leads to faster generalization. Source traces can be thought of as the model-based, backward view of successor representations (SR), and share many of the same benefits. This view, however, suggests several new ideas. First, a TD($λ$)-like source learning algorithm is proposed and its convergence is proven. Then, a novel algorithm for learning the source map (or SR matrix) is developed and shown to outperform the previous algorithm. Finally, various approaches to using the source/SR model are explored, and it is shown that source traces can be effectively combined with other model-based methods like Dyna and experience replay.