V2C-CBM: Building Concept Bottlenecks with Vision-to-Concept Tokenizer
This work addresses the problem of interpretability and efficiency in visual recognition for researchers and practitioners by offering a more accurate and training-efficient CBM, though it is incremental as it builds upon existing CBM and multimodal model frameworks.
The paper tackled the challenge of constructing interpretable Concept Bottleneck Models (CBMs) without relying on labor-intensive concept annotations or verbose language model outputs, by developing a Vision-to-Concept tokenizer that directly quantizes images into visual concepts using multimodal models, resulting in matched or superior accuracy on visual classification benchmarks compared to LLM-supervised CBMs.
Concept Bottleneck Models (CBMs) offer inherent interpretability by initially translating images into human-comprehensible concepts, followed by a linear combination of these concepts for classification. However, the annotation of concepts for visual recognition tasks requires extensive expert knowledge and labor, constraining the broad adoption of CBMs. Recent approaches have leveraged the knowledge of large language models to construct concept bottlenecks, with multimodal models like CLIP subsequently mapping image features into the concept feature space for classification. Despite this, the concepts produced by language models can be verbose and may introduce non-visual attributes, which hurts accuracy and interpretability. In this study, we investigate to avoid these issues by constructing CBMs directly from multimodal models. To this end, we adopt common words as base concept vocabulary and leverage auxiliary unlabeled images to construct a Vision-to-Concept (V2C) tokenizer that can explicitly quantize images into their most relevant visual concepts, thus creating a vision-oriented concept bottleneck tightly coupled with the multimodal model. This leads to our V2C-CBM which is training efficient and interpretable with high accuracy. Our V2C-CBM has matched or outperformed LLM-supervised CBMs on various visual classification benchmarks, validating the efficacy of our approach.