Refined Analysis of Entropy-Regularized Actor-Critic
Provides theoretical justification for the importance of accurate critic estimation in actor-critic methods for reinforcement learning.
The paper analyzes entropy-regularized actor-critic methods, proving that with an exact critic, actor-critic achieves $ ilde{O}(\log(1/\varepsilon))$ sample complexity for an $\varepsilon$-optimal regularized value, matching deterministic policy gradient. It also shows that variance reduction and rapid convergence are preserved when the critic error is sufficiently small.
In this paper, we study the role of the critic in actor--critic for entropy-regularized, finite, discounted environments. We establish that, when the critic is exact, using the latter as a baseline is a variance-reduction method in a strong sense. In this case, actor--critic with stochastic gradients matches the sample complexity of deterministic policy gradient, reaching an $ε$-optimal regularized value with $\tilde{O}(\log(1/ε))$ samples. In practice, the critic is learned alongside the actor: the variance of the actor update is then influenced by the critic's variance and bias. Specifically, when the critic has a sufficiently small error, the variance reduction and rapid convergence are preserved. This suggests to learn the critic first, keeping it up to date after each actor update, underscoring the crucial role of accurate critic estimation in actor--critic methods.