A Dynamic Penalty Function Approach for Constraints-Handling in Reinforcement Learning
This work offers an incremental improvement for researchers and practitioners applying reinforcement learning to constrained optimal control problems, specifically by mitigating training difficulties caused by large, static penalties.
This study addresses the computational issues arising from sharp changes in value functions at constraint boundaries when using soft penalty terms in reinforcement learning for constrained optimal control problems. The proposed dynamic penalty (DP) approach, which gradually increases the penalty factor during training, improves neural network approximation accuracy and provides faster convergence in a vehicle control problem.
Reinforcement learning (RL) is attracting attention as an effective way to solve sequential optimization problems that involve high dimensional state/action space and stochastic uncertainties. Many such problems involve constraints expressed by inequality constraints. This study focuses on using RL to solve constrained optimal control problems. Most RL application studies have dealt with inequality constraints by adding soft penalty terms for violating the constraints to the reward function. However, while training neural networks to learn the value (or Q) function, one can run into computational issues caused by the sharp change in the function value at the constraint boundary due to the large penalty imposed. This difficulty during training can lead to convergence problems and ultimately lead to poor closed-loop performance. To address this issue, this study proposes a dynamic penalty (DP) approach where the penalty factor is gradually and systematically increased during training as the iteration episodes proceed. We first examine the ability of a neural network to represent a value function when uniform, linear, or DP functions are added to prevent constraint violation. The agent trained by a Deep Q Network (DQN) algorithm with the DP function approach was compared with agents with other constant penalty functions in a simple vehicle control problem. Results show that the proposed approach can improve the neural network approximation accuracy and provide faster convergence when close to a solution.