Efficient Action-Constrained Reinforcement Learning via Acceptance-Rejection Method and Augmented MDPs
This addresses the need for safe and efficient control policies in safety-critical and resource-constrained applications, representing an incremental improvement over existing ACRL methods.
The paper tackles the problem of action-constrained reinforcement learning (ACRL) by proposing a framework that uses an acceptance-rejection method and augmented MDPs to enforce zero action constraint violations efficiently, resulting in faster training, better constraint satisfaction, and lower inference time compared to state-of-the-art methods.
Action-constrained reinforcement learning (ACRL) is a generic framework for learning control policies with zero action constraint violation, which is required by various safety-critical and resource-constrained applications. The existing ACRL methods can typically achieve favorable constraint satisfaction but at the cost of either high computational burden incurred by the quadratic programs (QP) or increased architectural complexity due to the use of sophisticated generative models. In this paper, we propose a generic and computationally efficient framework that can adapt a standard unconstrained RL method to ACRL through two modifications: (i) To enforce the action constraints, we leverage the classic acceptance-rejection method, where we treat the unconstrained policy as the proposal distribution and derive a modified policy with feasible actions. (ii) To improve the acceptance rate of the proposal distribution, we construct an augmented two-objective Markov decision process (MDP), which include additional self-loop state transitions and a penalty signal for the rejected actions. This augmented MDP incentives the learned policy to stay close to the feasible action sets. Through extensive experiments in both robot control and resource allocation domains, we demonstrate that the proposed framework enjoys faster training progress, better constraint satisfaction, and a lower action inference time simultaneously than the state-of-the-art ACRL methods. We have made the source code publicly available to encourage further research in this direction.