Cross-Modal Conceptualization in Bottleneck Models
This work addresses the difficulty of concept annotation in interpretable machine learning, particularly for medical imaging, though it is incremental as it builds on existing CBM frameworks.
The paper tackles the challenge of selecting and labeling concepts in Concept Bottleneck Models by using text descriptions to guide concept induction, demonstrating that this cross-modal approach encourages interpretable concepts and increases robustness by suppressing shortcut features.
Concept Bottleneck Models (CBMs) assume that training examples (e.g., x-ray images) are annotated with high-level concepts (e.g., types of abnormalities), and perform classification by first predicting the concepts, followed by predicting the label relying on these concepts. The main difficulty in using CBMs comes from having to choose concepts that are predictive of the label and then having to label training examples with these concepts. In our approach, we adopt a more moderate assumption and instead use text descriptions (e.g., radiology reports), accompanying the images in training, to guide the induction of concepts. Our cross-modal approach treats concepts as discrete latent variables and promotes concepts that (1) are predictive of the label, and (2) can be predicted reliably from both the image and text. Through experiments conducted on datasets ranging from synthetic datasets (e.g., synthetic images with generated descriptions) to realistic medical imaging datasets, we demonstrate that cross-modal learning encourages the induction of interpretable concepts while also facilitating disentanglement. Our results also suggest that this guidance leads to increased robustness by suppressing the reliance on shortcut features.