Central Limit Theorem for Two-Time-Scale Approximate Distributionally Robust RL
For researchers in robust RL, this provides a theoretically grounded, model-free algorithm with convergence guarantees, though it is limited to small-ambiguity regimes.
The paper tackles the challenges of model-free distributionally robust reinforcement learning by proposing an approximate framework based on a first-order expansion of the robust functional, leading to a model-free algorithm (MVSA) that converges with a central limit theorem at the canonical n^{-1/2} scale.
Designing model-free algorithms for distributionally robust reinforcement learning (DRRL) poses fundamental challenges. The robust Bellman operator is nonlinear in the transition kernel, which makes one-sample Bellman updates biased, while the adversarial optimization underlying robustness makes robust evaluation computationally demanding. To address these difficulties, we consider the natural small-ambiguity regime under Kullback--Leibler ambiguity sets and propose an approximate DRRL framework based on a first-order expansion of the relevant robust functional. This yields an approximate robust Bellman equation that removes the adversarial optimization while remaining first-order accurate in the ambiguity radius. To learn the fixed point of this approximate equation, we propose Mean-Variance Stochastic Approximation (MVSA), a model-free algorithm that uses only one-sample updates. This is achieved via a lifted stochastic approximation dynamics and a two-time-scale design. We then prove convergence and a central limit theorem for MVSA: its main iterate satisfies a central limit theorem at the canonical $n^{-1/2}$ scale, with explicitly characterized asymptotic covariances. Finally, we validate our theoretical findings with a numerical experiment.