TENG: Time-Evolving Natural Gradient for Solving PDEs With Deep Neural Nets Toward Machine Precision
This addresses the problem of persistent accuracy issues in neural-network-based PDE solvers for science and engineering applications, representing a strong specific gain rather than an incremental improvement.
The paper tackles the challenge of achieving high accuracy in solving partial differential equations (PDEs) with neural networks, particularly for initial value problems, by introducing the Time-Evolving Natural Gradient (TENG) method, which achieves machine precision in step-by-step optimizations across various PDEs like the heat equation and Burgers' equation.
Partial differential equations (PDEs) are instrumental for modeling dynamical systems in science and engineering. The advent of neural networks has initiated a significant shift in tackling these complexities though challenges in accuracy persist, especially for initial value problems. In this paper, we introduce the $\textit{Time-Evolving Natural Gradient (TENG)}$, generalizing time-dependent variational principles and optimization-based time integration, leveraging natural gradient optimization to obtain high accuracy in neural-network-based PDE solutions. Our comprehensive development includes algorithms like TENG-Euler and its high-order variants, such as TENG-Heun, tailored for enhanced precision and efficiency. TENG's effectiveness is further validated through its performance, surpassing current leading methods and achieving $\textit{machine precision}$ in step-by-step optimizations across a spectrum of PDEs, including the heat equation, Allen-Cahn equation, and Burgers' equation.