Convergence of Stochastic Gradient Methods for Wide Two-Layer Physics-Informed Neural Networks
This work provides theoretical guarantees for stochastic optimization methods in training PINNs, which is incremental as it extends prior results from gradient descent to stochastic methods.
The authors established linear convergence guarantees for stochastic gradient descent/flow when training over-parameterized two-layer physics-informed neural networks (PINNs) for a general class of activation functions, with results holding with high probability.
Physics informed neural networks (PINNs) represent a very popular class of neural solvers for partial differential equations. In practice, one often employs stochastic gradient descent type algorithms to train the neural network. Therefore, the convergence guarantee of stochastic gradient descent is of fundamental importance. In this work, we establish the linear convergence of stochastic gradient descent / flow in training over-parameterized two layer PINNs for a general class of activation functions in the sense of high probability. These results extend the existing result [18] in which gradient descent was analyzed. The challenge of the analysis lies in handling the dynamic randomness introduced by stochastic optimization methods. The key of the analysis lies in ensuring the positive definiteness of suitable Gram matrices during the training. The analysis sheds insight into the dynamics of the optimization process, and provides guarantees on the neural networks trained by stochastic algorithms.