VLMine: Long-Tail Data Mining with Vision Language Models
This addresses the challenge of long-tail data mining for applications like autonomous driving, offering a scalable method that integrates VLM signals with conventional techniques, though it appears incremental as it builds on existing VLM capabilities.
The paper tackles the problem of identifying rare examples in unlabeled data for robust performance on long-tail distributions, proposing a data mining approach using vision language models (VLMs) that shows improvements of 10% to 50% over baselines on benchmarks like ImageNet-LT, Places-LT, and Waymo Open Dataset.
Ensuring robust performance on long-tail examples is an important problem for many real-world applications of machine learning, such as autonomous driving. This work focuses on the problem of identifying rare examples within a corpus of unlabeled data. We propose a simple and scalable data mining approach that leverages the knowledge contained within a large vision language model (VLM). Our approach utilizes a VLM to summarize the content of an image into a set of keywords, and we identify rare examples based on keyword frequency. We find that the VLM offers a distinct signal for identifying long-tail examples when compared to conventional methods based on model uncertainty. Therefore, we propose a simple and general approach for integrating signals from multiple mining algorithms. We evaluate the proposed method on two diverse tasks: 2D image classification, in which inter-class variation is the primary source of data diversity, and on 3D object detection, where intra-class variation is the main concern. Furthermore, through the detection task, we demonstrate that the knowledge extracted from 2D images is transferable to the 3D domain. Our experiments consistently show large improvements (between 10\% and 50\%) over the baseline techniques on several representative benchmarks: ImageNet-LT, Places-LT, and the Waymo Open Dataset.