Implicit Stochastic Gradient Descent for Training Physics-informed Neural Networks
This addresses training instability in PINNs for solving differential equations, which is an incremental improvement for researchers in scientific computing and machine learning.
The paper tackles training failures in physics-informed neural networks (PINNs) when approximating high-frequency or multi-scale functions by proposing implicit stochastic gradient descent (ISGD) to improve stability, with empirical results showing it works well and compares favorably to methods like SGD and Adam.
Physics-informed neural networks (PINNs) have effectively been demonstrated in solving forward and inverse differential equation problems, but they are still trapped in training failures when the target functions to be approximated exhibit high-frequency or multi-scale features. In this paper, we propose to employ implicit stochastic gradient descent (ISGD) method to train PINNs for improving the stability of training process. We heuristically analyze how ISGD overcome stiffness in the gradient flow dynamics of PINNs, especially for problems with multi-scale solutions. We theoretically prove that for two-layer fully connected neural networks with large hidden nodes, randomly initialized ISGD converges to a globally optimal solution for the quadratic loss function. Empirical results demonstrate that ISGD works well in practice and compares favorably to other gradient-based optimization methods such as SGD and Adam, while can also effectively address the numerical stiffness in training dynamics via gradient descent.