Safe Reinforcement Learning of Control-Affine Systems with Vertex Networks
This addresses the challenge of ensuring safety in reinforcement learning for constrained control systems, which is crucial for real-world applications like robotics and autonomous systems, representing a novel method for a known bottleneck.
The paper tackled the problem of applying reinforcement learning to control systems with hard state and action constraints by proposing Vertex Networks, which incorporate safety constraints into the policy architecture to guarantee safe actions during exploration and in learned policies, outperforming vanilla reinforcement learning in benchmark control tasks.
This paper focuses on finding reinforcement learning policies for control systems with hard state and action constraints. Despite its success in many domains, reinforcement learning is challenging to apply to problems with hard constraints, especially if both the state variables and actions are constrained. Previous works seeking to ensure constraint satisfaction, or safety, have focused on adding a projection step to a learned policy. Yet, this approach requires solving an optimization problem at every policy execution step, which can lead to significant computational costs. To tackle this problem, this paper proposes a new approach, termed Vertex Networks (VNs), with guarantees on safety during exploration and on learned control policies by incorporating the safety constraints into the policy network architecture. Leveraging the geometric property that all points within a convex set can be represented as the convex combination of its vertices, the proposed algorithm first learns the convex combination weights and then uses these weights along with the pre-calculated vertices to output an action. The output action is guaranteed to be safe by construction. Numerical examples illustrate that the proposed VN algorithm outperforms vanilla reinforcement learning in a variety of benchmark control tasks.