Auto Cherry-Picker: Learning from High-quality Generative Data Driven by Language
This work addresses data scarcity and quality issues in perception tasks, offering a scalable solution for augmenting training data, though it is incremental as it builds on existing diffusion and LLM methods.
The paper tackles the challenge of using diffusion models to generate high-quality synthetic data for downstream perception tasks by introducing Auto Cherry-Picker (ACP), which uses LLMs and a controllable diffusion model to create and refine samples, resulting in significant performance improvements, especially for long-tailed and imbalanced datasets.
Diffusion models can generate realistic and diverse images, potentially facilitating data availability for data-intensive perception tasks. However, leveraging these models to boost performance on downstream tasks with synthetic data poses several challenges, including aligning with real data distribution, scaling synthetic sample volumes, and ensuring their quality. To bridge these gaps, we present \textbf{A}uto \textbf{C}herry-\textbf{P}icker (ACP), a novel framework that generates high-quality cross-modality training samples at scale to augment perception and multi-modal training. ACP first uses LLMs to sample descriptions and layouts based on object combinations from real data priors, eliminating the need for ground truth image captions or annotations. Next, we use an off-the-shelf controllable diffusion model to generate multiple images. Then, the generated data are refined using a comprehensively designed metric, Composite Layout and Image Score (CLIS), to ensure quality. Our customized synthetic high-quality samples boost performance in various scenarios, especially in addressing challenges associated with long-tailed distribution and imbalanced datasets. Experiment results on downstream tasks demonstrate that ACP can significantly improve the performance of existing models. In addition, we find a positive correlation between CLIS and performance gains in downstream tasks. This finding shows the potential for evaluation metrics as the role for various visual perception and MLLM tasks.