Learning Deep Hybrid Models with Sharpness-Aware Minimization
This addresses the challenge of effectively integrating scientific models with machine learning for robust and interpretable predictions, though it is incremental as it adapts an existing technique to a specific setting.
The paper tackles the problem of hybrid modeling, where machine learning models may ignore scientific components, by focusing on the flatness of loss minima using sharpness-aware minimization, resulting in a method that works well across various models and datasets.
Hybrid modeling, the combination of machine learning models and scientific mathematical models, enables flexible and robust data-driven prediction with partial interpretability. However, effectively the scientific models may be ignored in prediction due to the flexibility of the machine learning model, making the idea of hybrid modeling pointless. Typically some regularization is applied to hybrid model learning to avoid such a failure case, but the formulation of the regularizer strongly depends on model architectures and domain knowledge. In this paper, we propose to focus on the flatness of loss minima in learning hybrid models, aiming to make the model as simple as possible. We employ the idea of sharpness-aware minimization and adapt it to the hybrid modeling setting. Numerical experiments show that the SAM-based method works well across different choices of models and datasets.