Loss Landscape Analysis for Reliable Quantized ML Models for Scientific Sensing
This work addresses the problem of developing reliable machine learning models for scientific sensing applications, which is crucial for domains where experimental conditions are noisy and perturbations are common.
The authors tackled the problem of assessing the robustness of machine learning models for scientific sensing, finding a strong correlation between gently-shaped loss landscapes and robustness to input and weight perturbations. Their method allows for a systematic exploration of trade-offs between performance, efficiency, and robustness without requiring multiple model trainings.
In this paper, we propose a method to perform empirical analysis of the loss landscape of machine learning (ML) models. The method is applied to two ML models for scientific sensing, which necessitates quantization to be deployed and are subject to noise and perturbations due to experimental conditions. Our method allows assessing the robustness of ML models to such effects as a function of quantization precision and under different regularization techniques -- two crucial concerns that remained underexplored so far. By investigating the interplay between performance, efficiency, and robustness by means of loss landscape analysis, we both established a strong correlation between gently-shaped landscapes and robustness to input and weight perturbations and observed other intriguing and non-obvious phenomena. Our method allows a systematic exploration of such trade-offs a priori, i.e., without training and testing multiple models, leading to more efficient development workflows. This work also highlights the importance of incorporating robustness into the Pareto optimization of ML models, enabling more reliable and adaptive scientific sensing systems.