An Analysis of Quantile Temporal-Difference Learning
This provides theoretical grounding for QTD, which is used in large-scale reinforcement learning applications, but the work is incremental as it focuses on convergence analysis rather than new empirical gains.
The paper tackled the lack of theoretical understanding of quantile temporal-difference learning (QTD), a distributional reinforcement learning algorithm, by proving its convergence to fixed points of related dynamic programming procedures with probability 1.
We analyse quantile temporal-difference learning (QTD), a distributional reinforcement learning algorithm that has proven to be a key component in several successful large-scale applications of reinforcement learning. Despite these empirical successes, a theoretical understanding of QTD has proven elusive until now. Unlike classical TD learning, which can be analysed with standard stochastic approximation tools, QTD updates do not approximate contraction mappings, are highly non-linear, and may have multiple fixed points. The core result of this paper is a proof of convergence to the fixed points of a related family of dynamic programming procedures with probability 1, putting QTD on firm theoretical footing. The proof establishes connections between QTD and non-linear differential inclusions through stochastic approximation theory and non-smooth analysis.