Control invariant set enhanced reinforcement learning for process control: improved sampling efficiency and guaranteed stability
This work addresses safety and efficiency challenges in reinforcement learning for process control applications, representing an incremental advancement by integrating control invariant sets into RL training.
The paper tackled the problem of ensuring stability and improving sampling efficiency in reinforcement learning for process control by proposing a control invariant set enhanced RL approach, which resulted in significant improvements in sampling efficiency during offline training and guaranteed closed-loop stability in online implementation on a simulated chemical reactor.
Reinforcement learning (RL) is an area of significant research interest, and safe RL in particular is attracting attention due to its ability to handle safety-driven constraints that are crucial for real-world applications of RL algorithms. This work proposes a novel approach to RL training, called control invariant set (CIS) enhanced RL, which leverages the benefits of CIS to improve stability guarantees and sampling efficiency. The approach consists of two learning stages: offline and online. In the offline stage, CIS is incorporated into the reward design, initial state sampling, and state reset procedures. In the online stage, RL is retrained whenever the state is outside of CIS, which serves as a stability criterion. A backup table that utilizes the explicit form of CIS is obtained to ensure the online stability. To evaluate the proposed approach, we apply it to a simulated chemical reactor. The results show a significant improvement in sampling efficiency during offline training and closed-loop stability in the online implementation.