Manifold Metric: A Loss Landscape Approach for Predicting Model Performance
This work addresses the challenge of model selection and expansion for machine learning practitioners, offering a more efficient alternative to costly training, though it is incremental as it builds on existing loss landscape concepts.
The paper tackles the problem of efficiently predicting model performance for expansion of pre-trained models by introducing a loss landscape-based metric that estimates the size of a manifold of minima, revealing a strong correlation with performance gains and outperforming other baselines.
Determining the optimal model for a given task often requires training multiple models from scratch, which becomes impractical as dataset and model sizes grow. A more efficient alternative is to expand smaller pre-trained models, but this approach is underutilized due to a limited understanding of its impact on the training dynamics. Existing methods for quantifying this impact have notable limitations, including computation cost. To address this, we introduce a new perspective based on the loss landscape, which has been shown to contain a manifold of linearly connected minima. Specifically, we propose a metric that estimates the size of this manifold to study the impact of model expansion. Our experiments reveal a strong correlation between performance gains and our manifold metric, enabling more informed model comparison and offering a first step toward a geometry-driven approach for reliable model expansion. Notably, our metric outperforms other baselines, even when different types of expansion with equivalent number of parameters are applied to a model.