Benchmarking and Analyzing Generative Data for Visual Recognition
This addresses the need for better benchmarks and metrics to assess generative data in visual recognition, though it is incremental as it builds on existing generative models and retrieval paradigms.
This work tackles the problem of evaluating generative images for visual recognition by constructing GenBench, a broad benchmark with 22 datasets and 2548 categories, and proposing CLER, a training-free metric that correlates better with downstream performance than existing metrics like FID or CLIP score, leading to improved performance across 17 datasets through external knowledge injection.
Advancements in large pre-trained generative models have expanded their potential as effective data generators in visual recognition. This work delves into the impact of generative images, primarily comparing paradigms that harness external data (\ie generative \vs retrieval \vs original). Our key contributions are: \textbf{1) GenBench Construction:} We devise \textbf{GenBench}, a broad benchmark comprising 22 datasets with 2548 categories, to appraise generative data across various visual recognition tasks. \textbf{2) CLER Score:} To address the insufficient correlation of existing metrics (\eg, FID, CLIP score) with downstream recognition performance, we propose \textbf{CLER}, a training-free metric indicating generative data's efficiency for recognition tasks prior to training. \textbf{3) New Baselines:} Comparisons of generative data with retrieved data from the same external pool help to elucidate the unique traits of generative data. \textbf{4) External Knowledge Injection:} By fine-tuning special token embeddings for each category via Textual Inversion, performance improves across 17 datasets, except when dealing with low-resolution reference images. Our exhaustive benchmark and analysis spotlight generative data's promise in visual recognition, while identifying key challenges for future investigation.