PINNs Failure Modes are Overfitting
This work addresses a critical problem for researchers and practitioners using PINNs, explaining previously observed failure modes and offering a generalizable solution.
This paper investigates why Physics-Informed Neural Networks (PINNs) sometimes converge to incorrect solutions despite low loss. The authors demonstrate that these failures are due to overfitting, where the residual loss is minimized only at collocation points, and show that regularization can eliminate these failure modes. By extending double backpropagation, they achieve state-of-the-art performance on four standard failure mode equations, using up to 23 times fewer collocation points with a vanilla architecture.
Physics-Informed Neural Networks (PINNs) are a common class of machine learning-based partial differential equation (PDE) solvers which train a network to represent a solution by minimizing a residual loss that encodes the PDE. Despite their successes, they are known to fail on certain simple equations, converging to an incorrect solution despite low loss. These failure modes have garnered significant attention in the literature over the past several years, motivating both architectural and optimization based solutions. By directly visualizing the residual, we show that failure modes are the result of overfitting: the loss is minimized on the collocation points, but not elsewhere. Applying regularization causes the failure modes to vanish. Finally, we extend double backpropagation over the full set of residuals, and use it to achieve state-of-the-art performance on four standard failure mode equations with up to $23\times$ fewer collocation points and a vanilla architecture.