Selective Credit Assignment
This work addresses a fundamental challenge in reinforcement learning for improving algorithm efficiency, though it appears incremental as it builds on existing methods.
The paper tackles the problem of efficient credit assignment in reinforcement learning by introducing a unified view on temporal-difference algorithms that apply weightings to quantify contributions, enabling selective credit assignment including off-trajectory and off-policy scenarios.
Efficient credit assignment is essential for reinforcement learning algorithms in both prediction and control settings. We describe a unified view on temporal-difference algorithms for selective credit assignment. These selective algorithms apply weightings to quantify the contribution of learning updates. We present insights into applying weightings to value-based learning and planning algorithms, and describe their role in mediating the backward credit distribution in prediction and control. Within this space, we identify some existing online learning algorithms that can assign credit selectively as special cases, as well as add new algorithms that assign credit backward in time counterfactually, allowing credit to be assigned off-trajectory and off-policy.