Mitigating Numerical Stiffness in Least-Squares Formulations of Elliptic PDEs for Physics-Informed Neural Networks
For researchers using PINNs to solve elliptic PDEs, this work provides a principled method to mitigate training difficulties due to loss imbalance.
The paper analyzes $H^{-1}$ residual loss formulations for PINNs to address numerical stiffness caused by imbalanced loss terms in standard MSE formulations. Theoretical and experimental results show that $H^{-1}$ discretizations rebalance the PDE loss, improve conditioning, and mitigate stiffness, validated on Poisson and Navier-Stokes equations.
We present theoretical insights into $H^{-1}$ residual loss formulations of physics-informed neural networks (PINNs) for learning solutions of partial differential equations (PDEs). Standard PINN formulations use a multi-term loss functional consisting of interior and boundary loss terms that are based on $L^2$-residuals and discretized as mean square errors (MSE). Imbalanced magnitudes of these terms cause numerical stiffness phenomena, resulting in ill-conditioning and slow convergence. In this work, we analyze discretizations of the $H^{-1}$-norm that are used in the context of elliptic PDEs with arbitrary, nonzero Dirichlet boundary conditions. We prove that these $H^{-1}$ discretizations rebalance the PDE loss, improve conditioning, and mitigate stiffness effects compared with the standard MSE discretization. We validate our theoretic results through operator-level experiments with randomly sampled residuals and PINN experiments for the Poisson and stationary incompressible Navier-Stokes equations. These experiments confirm the numerical effectiveness of the proposed rebalancing for elliptic PDEs and, more broadly, for problems with elliptic behavior.