Finite-Sample Analysis of Off-Policy Natural Actor-Critic Algorithm
This addresses the challenge of sample efficiency and variance control in reinforcement learning for practitioners, though it is incremental as it builds on existing methods like V-trace.
The paper tackles the problem of providing finite-sample convergence guarantees for an off-policy natural actor-critic algorithm, achieving a sample complexity of O(ε^{-3} log^2(1/ε)) to converge to a global optimal policy.
In this paper, we provide finite-sample convergence guarantees for an off-policy variant of the natural actor-critic (NAC) algorithm based on Importance Sampling. In particular, we show that the algorithm converges to a global optimal policy with a sample complexity of $\mathcal{O}(ε^{-3}\log^2(1/ε))$ under an appropriate choice of stepsizes. In order to overcome the issue of large variance due to Importance Sampling, we propose the $Q$-trace algorithm for the critic, which is inspired by the V-trace algorithm \cite{espeholt2018impala}. This enables us to explicitly control the bias and variance, and characterize the trade-off between them. As an advantage of off-policy sampling, a major feature of our result is that we do not need any additional assumptions, beyond the ergodicity of the Markov chain induced by the behavior policy.