Momentum Diminishes the Effect of Spectral Bias in Physics-Informed Neural Networks
This addresses a key bottleneck for researchers and practitioners using PINNs to solve complex PDEs, though it is incremental as it builds on existing methods like NTK analysis.
The paper tackles the problem of spectral bias in physics-informed neural networks (PINNs), which causes failure when solving PDEs with high-frequency features, and shows that using stochastic gradient descent with momentum (SGDM) significantly reduces this effect, enabling convergence to desirable solutions even with high-frequency features, as confirmed by numerical experiments.
Physics-informed neural network (PINN) algorithms have shown promising results in solving a wide range of problems involving partial differential equations (PDEs). However, they often fail to converge to desirable solutions when the target function contains high-frequency features, due to a phenomenon known as spectral bias. In the present work, we exploit neural tangent kernels (NTKs) to investigate the training dynamics of PINNs evolving under stochastic gradient descent with momentum (SGDM). This demonstrates SGDM significantly reduces the effect of spectral bias. We have also examined why training a model via the Adam optimizer can accelerate the convergence while reducing the spectral bias. Moreover, our numerical experiments have confirmed that wide-enough networks using SGDM still converge to desirable solutions, even in the presence of high-frequency features. In fact, we show that the width of a network plays a critical role in convergence.