A Convergence Result for Regularized Actor-Critic Methods
This provides theoretical guarantees for reinforcement learning methods, but it is incremental as it extends existing convergence proofs to regularized settings.
The paper tackles the problem of proving convergence for actor-critic algorithms in entropy-regularized Markov decision processes, and it establishes a probability one convergence result under suitable conditions.
In this paper, we present a probability one convergence proof, under suitable conditions, of a certain class of actor-critic algorithms for finding approximate solutions to entropy-regularized MDPs using the machinery of stochastic approximation. To obtain this overall result, we prove the convergence of policy evaluation with general regularizers when using linear approximation architectures and show convergence of entropy-regularized policy improvement.