Variational Actor-Critic Algorithms
This work addresses convergence issues in reinforcement learning algorithms, offering incremental improvements for researchers and practitioners in the field.
The paper tackles the problem of improving actor-critic algorithms by introducing a variational formulation that maximizes the value function and minimizes the Bellman residual, resulting in faster convergence through proposed variants like clipping and flipping methods, with a proof that the fixed point approaches the optimal policy under certain conditions.
We introduce a class of variational actor-critic algorithms based on a variational formulation over both the value function and the policy. The objective function of the variational formulation consists of two parts: one for maximizing the value function and the other for minimizing the Bellman residual. Besides the vanilla gradient descent with both the value function and the policy updates, we propose two variants, the clipping method and the flipping method, in order to speed up the convergence. We also prove that, when the prefactor of the Bellman residual is sufficiently large, the fixed point of the algorithm is close to the optimal policy.