Sequential-in-time training of nonlinear parametrizations for solving time-dependent partial differential equations
This provides theoretical insights for researchers in numerical analysis and machine learning working on PDE solvers, though it appears incremental as it unifies existing concepts rather than introducing a new paradigm.
The paper tackles the problem of training nonlinear parametrizations like neural networks to approximate solutions of time-dependent partial differential equations, showing that sequential-in-time methods can be categorized as optimize-then-discretize or discretize-then-optimize schemes, leading to new stability and error analysis results.
Sequential-in-time methods solve a sequence of training problems to fit nonlinear parametrizations such as neural networks to approximate solution trajectories of partial differential equations over time. This work shows that sequential-in-time training methods can be understood broadly as either optimize-then-discretize (OtD) or discretize-then-optimize (DtO) schemes, which are well known concepts in numerical analysis. The unifying perspective leads to novel stability and a posteriori error analysis results that provide insights into theoretical and numerical aspects that are inherent to either OtD or DtO schemes such as the tangent space collapse phenomenon, which is a form of over-fitting. Additionally, the unified perspective facilitates establishing connections between variants of sequential-in-time training methods, which is demonstrated by identifying natural gradient descent methods on energy functionals as OtD schemes applied to the corresponding gradient flows.