Self-Evolving Visual Concept Library using Vision-Language Critics
This addresses the challenge of creating effective visual concept libraries for visual recognition, offering an automated solution that could benefit researchers and practitioners in computer vision.
The paper tackles the problem of building visual concept libraries for recognition by proposing ESCHER, an automated framework that iteratively refines concepts using a vision-language model critic, resulting in improved performance for zero-shot, few-shot, and fine-tuning classification tasks without human annotations.
We study the problem of building a visual concept library for visual recognition. Building effective visual concept libraries is challenging, as manual definition is labor-intensive, while relying solely on LLMs for concept generation can result in concepts that lack discriminative power or fail to account for the complex interactions between them. Our approach, ESCHER, takes a library learning perspective to iteratively discover and improve visual concepts. ESCHER uses a vision-language model (VLM) as a critic to iteratively refine the concept library, including accounting for interactions between concepts and how they affect downstream classifiers. By leveraging the in-context learning abilities of LLMs and the history of performance using various concepts, ESCHER dynamically improves its concept generation strategy based on the VLM critic's feedback. Finally, ESCHER does not require any human annotations, and is thus an automated plug-and-play framework. We empirically demonstrate the ability of ESCHER to learn a concept library for zero-shot, few-shot, and fine-tuning visual classification tasks. This work represents, to our knowledge, the first application of concept library learning to real-world visual tasks.