Reinforcement learning-based estimation for partial differential equations
This work addresses the challenge of accurate state estimation in fluid dynamics and similar systems for researchers and engineers, representing an incremental improvement by integrating reinforcement learning into existing ROM-based frameworks.
The authors tackled the problem of state estimation in systems governed by nonlinear partial differential equations, where reduced-order models (ROMs) often lead to large errors, by introducing a reinforcement learning-based estimator (RL-ROE) that uses a nonlinear policy to compensate for ROM errors. They demonstrated that RL-ROE outperforms a Kalman filter using the same ROM in scenarios with very few sensors and yields accurate state estimates across various physical parameters without direct knowledge of them.
In systems governed by nonlinear partial differential equations such as fluid flows, the design of state estimators such as Kalman filters relies on a reduced-order model (ROM) that projects the original high-dimensional dynamics onto a computationally tractable low-dimensional space. However, ROMs are prone to large errors, which negatively affects the performance of the estimator. Here, we introduce the reinforcement learning reduced-order estimator (RL-ROE), a ROM-based estimator in which the correction term that takes in the measurements is given by a nonlinear policy trained through reinforcement learning. The nonlinearity of the policy enables the RL-ROE to compensate efficiently for errors of the ROM, while still taking advantage of the imperfect knowledge of the dynamics. Using examples involving the Burgers and Navier-Stokes equations, we show that in the limit of very few sensors, the trained RL-ROE outperforms a Kalman filter designed using the same ROM. Moreover, it yields accurate high-dimensional state estimates for trajectories corresponding to various physical parameter values, without direct knowledge of the latter.