Adaptive Trade-Offs in Off-Policy Learning
This work addresses the challenge of algorithm selection and efficiency in reinforcement learning for researchers and practitioners, though it is incremental as it builds on existing methods.
The paper tackles the problem of unifying off-policy learning algorithms by analyzing trade-offs among update variance, fixed-point bias, and contraction rate, resulting in the novel C-trace algorithm that achieves state-of-the-art performance in large-scale environments.
A great variety of off-policy learning algorithms exist in the literature, and new breakthroughs in this area continue to be made, improving theoretical understanding and yielding state-of-the-art reinforcement learning algorithms. In this paper, we take a unifying view of this space of algorithms, and consider their trade-offs of three fundamental quantities: update variance, fixed-point bias, and contraction rate. This leads to new perspectives of existing methods, and also naturally yields novel algorithms for off-policy evaluation and control. We develop one such algorithm, C-trace, demonstrating that it is able to more efficiently make these trade-offs than existing methods in use, and that it can be scaled to yield state-of-the-art performance in large-scale environments.