Enhancing Zero-Shot Image Recognition in Vision-Language Models through Human-like Concept Guidance
This addresses the challenge of improving zero-shot generalization in vision-language models for real-world applications, representing a strong specific gain rather than a foundational breakthrough.
The paper tackles the problem of sub-optimal performance in zero-shot image recognition by vision-language models due to poor prompt engineering and adaptation, proposing a Concept-guided Human-like Bayesian Reasoning (CHBR) framework that outperforms state-of-the-art methods across fifteen datasets.
In zero-shot image recognition tasks, humans demonstrate remarkable flexibility in classifying unseen categories by composing known simpler concepts. However, existing vision-language models (VLMs), despite achieving significant progress through large-scale natural language supervision, often underperform in real-world applications because of sub-optimal prompt engineering and the inability to adapt effectively to target classes. To address these issues, we propose a Concept-guided Human-like Bayesian Reasoning (CHBR) framework. Grounded in Bayes' theorem, CHBR models the concept used in human image recognition as latent variables and formulates this task by summing across potential concepts, weighted by a prior distribution and a likelihood function. To tackle the intractable computation over an infinite concept space, we introduce an importance sampling algorithm that iteratively prompts large language models (LLMs) to generate discriminative concepts, emphasizing inter-class differences. We further propose three heuristic approaches involving Average Likelihood, Confidence Likelihood, and Test Time Augmentation (TTA) Likelihood, which dynamically refine the combination of concepts based on the test image. Extensive evaluations across fifteen datasets demonstrate that CHBR consistently outperforms existing state-of-the-art zero-shot generalization methods.