Actor-Accelerated Policy Dual Averaging for Reinforcement Learning in Continuous Action Spaces
This work addresses the practical deployment of PDA for reinforcement learning in continuous-action problems, representing an incremental improvement by accelerating an existing method.
The paper tackles the computational challenge of applying Policy Dual Averaging (PDA) in continuous action spaces by proposing actor-accelerated PDA, which uses a learned policy network to approximate optimization sub-problems, achieving superior performance compared to baselines like Proximal Policy Optimization (PPO) on robotics, control, and operations research benchmarks.
Policy Dual Averaging (PDA) offers a principled Policy Mirror Descent (PMD) framework that more naturally admits value function approximation than standard PMD, enabling the use of approximate advantage (or Q-) functions while retaining strong convergence guarantees. However, applying PDA in continuous state and action spaces remains computationally challenging, since action selection involves solving an optimization sub-problem at each decision step. In this paper, we propose \textit{actor-accelerated PDA}, which uses a learned policy network to approximate the solution of the optimization sub-problems, yielding faster runtimes while maintaining convergence guarantees. We provide a theoretical analysis that quantifies how actor approximation error impacts the convergence of PDA under suitable assumptions. We then evaluate its performance on several benchmarks in robotics, control, and operations research problems. Actor-accelerated PDA achieves superior performance compared to popular on-policy baselines such as Proximal Policy Optimization (PPO). Overall, our results bridge the gap between the theoretical advantages of PDA and its practical deployment in continuous-action problems with function approximation.