DataRater: Meta-Learned Dataset Curation
This addresses the challenge of scalable and effective dataset curation for AI researchers and practitioners, offering a novel approach beyond manual tuning or heuristic-based filtering.
The paper tackles the problem of dataset curation for foundation models by proposing DataRater, a meta-learning method that estimates the value of individual data points to improve training efficiency, resulting in significantly improved compute efficiency across various model scales and datasets.
The quality of foundation models depends heavily on their training data. Consequently, great efforts have been put into dataset curation. Yet most approaches rely on manual tuning of coarse-grained mixtures of large buckets of data, or filtering by hand-crafted heuristics. An approach that is ultimately more scalable (let alone more satisfying) is to \emph{learn} which data is actually valuable for training. This type of meta-learning could allow more sophisticated, fine-grained, and effective curation. Our proposed \emph{DataRater} is an instance of this idea. It estimates the value of training on any particular data point. This is done by meta-learning using `meta-gradients', with the objective of improving training efficiency on held out data. In extensive experiments across a range of model scales and datasets, we find that using our DataRater to filter data is highly effective, resulting in significantly improved compute efficiency.