Refining embeddings with fill-tuning: data-efficient generalised performance improvements for materials foundation models
This addresses the trade-off between task-specific performance and generalizability in foundation models, offering a computationally efficient method for broad improvements, though it appears incremental as an enhancement to existing fine-tuning approaches.
The paper tackles the problem of foundation models degrading performance on out-of-distribution tasks when fine-tuned for specific tasks, by introducing 'fill-tuning' to generate datasets that correct poor embedding regions instead. The result shows almost 1% improvement in all downstream tasks for materials foundation models with only 100 additional data points.
Pretrained foundation models learn embeddings that can be used for a wide range of downstream tasks. These embeddings optimise general performance, and if insufficiently accurate at a specific task the model can be fine-tuned to improve performance. For all current methodologies this operation necessarily degrades performance on all out-of-distribution tasks. In this work we present 'fill-tuning', a novel methodology to generate datasets for continued pretraining of foundation models that are not suited to a particular downstream task, but instead aim to correct poor regions of the embedding. We present the application of roughness analysis to latent space topologies and illustrate how it can be used to propose data that will be most valuable to improving the embedding. We apply fill-tuning to a set of state-of-the-art materials foundation models trained on $O(10^9)$ data points and show model improvement of almost 1% in all downstream tasks with the addition of only 100 data points. This method provides a route to the general improvement of foundation models at the computational cost of fine-tuning.