Policy Optimization for PDE Control with a Warm Start
This addresses the challenge of controlling chaotic PDEs with reduced-order models, offering a cost-effective alternative to end-to-end reinforcement learning, though it is incremental as it augments an existing strategy.
The paper tackles the problem of controller performance degradation due to inaccuracies in reduced-order modeling for PDE control, showing that a few iterations of policy optimization can significantly improve the model-based controller performance.
Dimensionality reduction is crucial for controlling nonlinear partial differential equations (PDE) through a "reduce-then-design" strategy, which identifies a reduced-order model and then implements model-based control solutions. However, inaccuracies in the reduced-order modeling can substantially degrade controller performance, especially in PDEs with chaotic behavior. To address this issue, we augment the reduce-then-design procedure with a policy optimization (PO) step. The PO step fine-tunes the model-based controller to compensate for the modeling error from dimensionality reduction. This augmentation shifts the overall strategy into reduce-then-design-then-adapt, where the model-based controller serves as a warm start for PO. Specifically, we study the state-feedback tracking control of PDEs that aims to align the PDE state with a specific constant target subject to a linear-quadratic cost. Through extensive experiments, we show that a few iterations of PO can significantly improve the model-based controller performance. Our approach offers a cost-effective alternative to PDE control using end-to-end reinforcement learning.