Escaping from Zero Gradient: Revisiting Action-Constrained Reinforcement Learning via Frank-Wolfe Policy Optimization
This addresses a critical bottleneck in real-world RL applications like robotics and scheduling, offering improved efficiency and convergence, though it is an incremental advancement over existing projection-based methods.
The paper tackles the zero-gradient problem in action-constrained reinforcement learning, which causes sample-inefficient training, by proposing a Frank-Wolfe policy optimization algorithm that decouples constraints from policy updates. The result is a method that significantly outperforms benchmarks on various control tasks, with concrete performance gains demonstrated experimentally.
Action-constrained reinforcement learning (RL) is a widely-used approach in various real-world applications, such as scheduling in networked systems with resource constraints and control of a robot with kinematic constraints. While the existing projection-based approaches ensure zero constraint violation, they could suffer from the zero-gradient problem due to the tight coupling of the policy gradient and the projection, which results in sample-inefficient training and slow convergence. To tackle this issue, we propose a learning algorithm that decouples the action constraints from the policy parameter update by leveraging state-wise Frank-Wolfe and a regression-based policy update scheme. Moreover, we show that the proposed algorithm enjoys convergence and policy improvement properties in the tabular case as well as generalizes the popular DDPG algorithm for action-constrained RL in the general case. Through experiments, we demonstrate that the proposed algorithm significantly outperforms the benchmark methods on a variety of control tasks.