ConFIG: Towards Conflict-free Training of Physics Informed Neural Networks
This addresses the difficulty of training PINNs, a domain-specific problem in scientific machine learning, with incremental improvements to optimization methods.
The paper tackles the problem of conflicting loss terms in Physics-Informed Neural Networks (PINNs) by proposing the ConFIG method, which ensures conflict-free updates and consistent optimization rates, resulting in superior performance and runtime compared to baselines across challenging PINN scenarios.
The loss functions of many learning problems contain multiple additive terms that can disagree and yield conflicting update directions. For Physics-Informed Neural Networks (PINNs), loss terms on initial/boundary conditions and physics equations are particularly interesting as they are well-established as highly difficult tasks. To improve learning the challenging multi-objective task posed by PINNs, we propose the ConFIG method, which provides conflict-free updates by ensuring a positive dot product between the final update and each loss-specific gradient. It also maintains consistent optimization rates for all loss terms and dynamically adjusts gradient magnitudes based on conflict levels. We additionally leverage momentum to accelerate optimizations by alternating the back-propagation of different loss terms. We provide a mathematical proof showing the convergence of the ConFIG method, and it is evaluated across a range of challenging PINN scenarios. ConFIG consistently shows superior performance and runtime compared to baseline methods. We also test the proposed method in a classic multi-task benchmark, where the ConFIG method likewise exhibits a highly promising performance. Source code is available at https://tum-pbs.github.io/ConFIG