Accelerating Primal-dual Methods for Regularized Markov Decision Processes
This work addresses a bottleneck in reinforcement learning for researchers and practitioners, though it is incremental as it builds on existing primal-dual frameworks.
The paper tackled the slow convergence of primal-dual methods in entropy-regularized Markov decision processes by introducing a quadratically convexified formulation and an interpolating metric, achieving exponential convergence rates and significant acceleration in numerical tests.
Entropy regularized Markov decision processes have been widely used in reinforcement learning. This paper is concerned with the primal-dual formulation of the entropy regularized problems. Standard first-order methods suffer from slow convergence due to the lack of strict convexity and concavity. To address this issue, we first introduce a new quadratically convexified primal-dual formulation. The natural gradient ascent descent of the new formulation enjoys global convergence guarantee and exponential convergence rate. We also propose a new interpolating metric that further accelerates the convergence significantly. Numerical results are provided to demonstrate the performance of the proposed methods under multiple settings.