Evaluating Sample Utility for Efficient Data Selection by Mimicking Model Weights
This work addresses the challenge of selecting high-quality training data for multimodal models, offering an efficient alternative to existing methods, though it is incremental as it builds on gradient-based techniques.
The paper tackles the problem of inefficient and noisy training data for multimodal models by proposing the Mimic Score, a model-based data-quality metric that uses reference model weights to assess sample utility, resulting in improved data efficiency, faster convergence, and performance gains across six image datasets, including a 20.7% reduction in training steps for CLIP models.
Multimodal models are trained on large-scale web-crawled datasets, which often contain noise, bias, and irrelevant information. This motivates the use of data selection techniques, which can be divided into model-free variants, relying on heuristic rules and downstream datasets, and model-based approaches, such as those using influence functions. The former can be expensive to design and risks introducing unwanted dataset dependencies, while the latter are often computationally prohibitive. In this work, we propose an efficient, model-based approach using the Mimic Score, a new data-quality metric that leverages the weights of a reference model to assess the usefulness of individual samples for training a new model. Our method relies on measuring alignments between training gradients and a target direction induced by this reference model. Building on the derived mimic scores, we develop Grad-Mimic: a framework that prioritizes samples to learn, estimates overall sample utility, and creates effective filters. Empirically, using mimic scores to guide training improves data efficiency, accelerates convergence, yields consistent performance gains across six image datasets, and enhances CLIP models with 20.7% fewer training steps. Moreover, mimic score-based filters complement existing filtering methods, e.g., training improved CLIP models with 4.7 million fewer samples while offering accurate estimation of dataset quality.