Neural Operator based Reinforcement Learning for Control of first-order PDEs with Spatially-Varying State Delay
This addresses control challenges in distributed parameter systems with spatial delays, offering a hybrid approach that is incremental but improves performance in specific applications.
The paper tackled controlling unstable first-order hyperbolic PDEs with spatially-varying delays by integrating PDE backstepping control with deep reinforcement learning, using a SAC architecture with DeepONet to approximate the controller, and in simulations, it outperformed baseline SAC and analytical controllers.
Control of distributed parameter systems affected by delays is a challenging task, particularly when the delays depend on spatial variables. The idea of integrating analytical control theory with learning-based control within a unified control scheme is becoming increasingly promising and advantageous. In this paper, we address the problem of controlling an unstable first-order hyperbolic PDE with spatially-varying delays by combining PDE backstepping control strategies and deep reinforcement learning (RL). To eliminate the assumption on the delay function required for the backstepping design, we propose a soft actor-critic (SAC) architecture incorporating a DeepONet to approximate the backstepping controller. The DeepONet extracts features from the backstepping controller and feeds them into the policy network. In simulations, our algorithm outperforms the baseline SAC without prior backstepping knowledge and the analytical controller.