Towards Efficient and General-Purpose Few-Shot Misclassification Detection for Vision-Language Models
This work addresses the need for reliable confidence estimation in classifiers for high-security and dynamic applications, offering an incremental improvement over existing methods by enhancing efficiency and generalizability.
The paper tackles the problem of misclassification detection in vision-language models by proposing a few-shot prompt learning framework that avoids training from scratch, achieving significant and consistent improvements in efficiency and generalization across datasets with domain shifts.
Reliable prediction by classifiers is crucial for their deployment in high security and dynamically changing situations. However, modern neural networks often exhibit overconfidence for misclassified predictions, highlighting the need for confidence estimation to detect errors. Despite the achievements obtained by existing methods on small-scale datasets, they all require training from scratch and there are no efficient and effective misclassification detection (MisD) methods, hindering practical application towards large-scale and ever-changing datasets. In this paper, we pave the way to exploit vision language model (VLM) leveraging text information to establish an efficient and general-purpose misclassification detection framework. By harnessing the power of VLM, we construct FSMisD, a Few-Shot prompt learning framework for MisD to refrain from training from scratch and therefore improve tuning efficiency. To enhance misclassification detection ability, we use adaptive pseudo sample generation and a novel negative loss to mitigate the issue of overconfidence by pushing category prompts away from pseudo features. We conduct comprehensive experiments with prompt learning methods and validate the generalization ability across various datasets with domain shift. Significant and consistent improvement demonstrates the effectiveness, efficiency and generalizability of our approach.