Introduction to optimization methods for training SciML models
This is an incremental introduction for researchers and practitioners in SciML, focusing on how problem structure influences algorithmic choices without presenting new methods or results.
The paper addresses the distinct optimization challenges in scientific machine learning (SciML) compared to classical machine learning, highlighting that SciML's physics-informed formulations lead to global coupling and stiffness, which limit standard stochastic methods and require deterministic or curvature-aware approaches.
Optimization is central to both modern machine learning (ML) and scientific machine learning (SciML), yet the structure of the underlying optimization problems differs substantially across these domains. Classical ML typically relies on stochastic, sample-separable objectives that favor first-order and adaptive gradient methods. In contrast, SciML often involves physics-informed or operator-constrained formulations in which differential operators induce global coupling, stiffness, and strong anisotropy in the loss landscape. As a result, optimization behavior in SciML is governed by the spectral properties of the underlying physical models rather than by data statistics, frequently limiting the effectiveness of standard stochastic methods and motivating deterministic or curvature-aware approaches. This document provides a unified introduction to optimization methods in ML and SciML, emphasizing how problem structure shapes algorithmic choices. We review first- and second-order optimization techniques in both deterministic and stochastic settings, discuss their adaptation to physics-constrained and data-driven SciML models, and illustrate practical strategies through tutorial examples, while highlighting open research directions at the interface of scientific computing and scientific machine learning.