The Devil is in the Details: A Deep Dive into the Rabbit Hole of Data Filtering
This work addresses the problem of data filtering for foundation models, which is crucial for AI researchers and practitioners, though it is incremental as it builds on existing methods.
The paper tackles the challenge of evaluating and improving data filtering methods for foundation models by participating in the DataComp benchmark, achieving over 4% better average performance on 38 tasks and over 2% improvement on ImageNet compared to prior methods.
The quality of pre-training data plays a critical role in the performance of foundation models. Popular foundation models often design their own recipe for data filtering, which makes it hard to analyze and compare different data filtering approaches. DataComp is a new benchmark dedicated to evaluating different methods for data filtering. This paper describes our learning and solution when participating in the DataComp challenge. Our filtering strategy includes three stages: single-modality filtering, cross-modality filtering, and data distribution alignment. We integrate existing methods and propose new solutions, such as computing CLIP score on horizontally flipped images to mitigate the interference of scene text, using vision and language models to retrieve training samples for target downstream tasks, rebalancing the data distribution to improve the efficiency of allocating the computational budget, etc. We slice and dice our design choices, provide in-depth analysis, and discuss open questions. Our approach outperforms the best method from the DataComp paper by over 4% on the average performance of 38 tasks and by over 2% on ImageNet.