Unveiling Multi-regime Patterns in SciML: Distinct Failure Modes and Regime-specific Optimization
For practitioners of scientific machine learning, this work offers a diagnostic tool to understand and mitigate regime-specific training failures, though the findings are primarily observational and incremental.
The paper identifies a consistent three-regime structure in scientific machine learning models, showing that optimization effectiveness is regime-specific and that standard loss-landscape metrics can be misleading. These findings provide a framework for diagnosing failure modes and improving robustness across models like PINNs, neural operators, and neural ODEs.
Neural networks trained under different hyperparameter settings can fall into distinct training "regimes," with consistent behavior within regimes and qualitative differences across regimes. In this paper, we study such multi-regime behavior in scientific machine learning (SciML) models through a regime-aware diagnostic framework that jointly analyzes performance, training dynamics, and loss-landscape geometry. We identify three key findings: (i) a consistent three-regime structure emerges across many standard SciML models, different constraint enforcements, and various optimizer designs; (ii) optimization effectiveness is regime-specific, with no single method performing well across all regimes; and (iii) SciML models can exhibit fine-grained failure modes that can challenge conventional interpretations of standard loss-landscape metrics. Our results provide an approach to establish a unified, task-oblivious perspective on failure modes in SciML and to inform regime-aware guidance for improving robustness. We validate these findings across widely-used SciML models, including physics-informed neural networks, neural operators, and neural ordinary differential equations, on benchmarks spanning representative ordinary and partial differential equations.