Central Limit Theorems for Asynchronous Averaged Q-Learning
This provides theoretical guarantees for the convergence behavior of asynchronous Q-learning algorithms, which is incremental but important for reinforcement learning practitioners.
The paper tackles the problem of establishing central limit theorems for Polyak-Ruppert averaged Q-learning under asynchronous updates, proving a non-asymptotic central limit theorem with explicit convergence rates and a functional central limit theorem showing weak convergence to Brownian motion.
This paper establishes central limit theorems for Polyak-Ruppert averaged Q-learning under asynchronous updates. We prove a non-asymptotic central limit theorem, where the convergence rate in Wasserstein distance explicitly reflects the dependence on the number of iterations, state-action space size, the discount factor, and the quality of exploration. In addition, we derive a functional central limit theorem, showing that the partial-sum process converges weakly to a Brownian motion.