AutoBalance: An Automatic Balancing Framework for Training Physics-Informed Neural Networks
This addresses a critical training difficulty for researchers and practitioners using PINNs to solve partial differential equations, offering an incremental improvement over existing methods.
The paper tackles the challenge of balancing multiple loss terms in Physics-Informed Neural Networks (PINNs), which often have conflicting objectives and different curvatures, by introducing AutoBalance, a novel 'post-combine' training paradigm that assigns independent adaptive optimizers to each loss component. The result is a significant reduction in solution error, as shown by MSE and L∞ norms on challenging PDE benchmarks.
Physics-Informed Neural Networks (PINNs) provide a powerful and general framework for solving Partial Differential Equations (PDEs) by embedding physical laws into loss functions. However, training PINNs is notoriously difficult due to the need to balance multiple loss terms, such as PDE residuals and boundary conditions, which often have conflicting objectives and vastly different curvatures. Existing methods address this issue by manipulating gradients before optimization (a "pre-combine" strategy). We argue that this approach is fundamentally limited, as forcing a single optimizer to process gradients from spectrally heterogeneous loss landscapes disrupts its internal preconditioning. In this work, we introduce AutoBalance, a novel "post-combine" training paradigm. AutoBalance assigns an independent adaptive optimizer to each loss component and aggregates the resulting preconditioned updates afterwards. Extensive experiments on challenging PDE benchmarks show that AutoBalance consistently outperforms existing frameworks, achieving significant reductions in solution error, as measured by both the MSE and $L^{\infty}$ norms. Moreover, AutoBalance is orthogonal to and complementary with other popular PINN methodologies, amplifying their effectiveness on demanding benchmarks.