Precision Machine Learning
This work addresses optimization challenges in high-precision ML for scientific applications, but it is incremental as it focuses on training improvements rather than a fundamental breakthrough.
The paper tackles the challenge of training machine learning models to achieve very high precision, particularly for scientific applications, by comparing function approximation methods and finding neural networks excel in high-dimensional cases but struggle in low-dimensional ones. To address this, the authors develop training tricks that enable neural networks to reach extremely low loss, near numerical precision limits.
We explore unique considerations involved in fitting ML models to data with very high precision, as is often required for science applications. We empirically compare various function approximation methods and study how they scale with increasing parameters and data. We find that neural networks can often outperform classical approximation methods on high-dimensional examples, by auto-discovering and exploiting modular structures therein. However, neural networks trained with common optimizers are less powerful for low-dimensional cases, which motivates us to study the unique properties of neural network loss landscapes and the corresponding optimization challenges that arise in the high precision regime. To address the optimization issue in low dimensions, we develop training tricks which enable us to train neural networks to extremely low loss, close to the limits allowed by numerical precision.