TextCAM: Explaining Class Activation Map with Text
This addresses the interpretability issue in deep neural networks for vision applications, enhancing trustworthiness in high-stakes scenarios, though it is an incremental improvement over existing CAM methods.
The paper tackled the problem that Class Activation Mapping (CAM) methods for explaining deep vision models lack semantic insight into what attributes drive predictions, and proposed TextCAM, a framework that enriches CAM with natural language descriptions using vision-language models, resulting in explanations that improve human understanding and detect spurious correlations in experiments on datasets like ImageNet.
Deep neural networks (DNNs) have achieved remarkable success across domains but remain difficult to interpret, limiting their trustworthiness in high-stakes applications. This paper focuses on deep vision models, for which a dominant line of explainability methods are Class Activation Mapping (CAM) and its variants working by highlighting spatial regions that drive predictions. We figure out that CAM provides little semantic insight into what attributes underlie these activations. To address this limitation, we propose TextCAM, a novel explanation framework that enriches CAM with natural languages. TextCAM combines the precise spatial localization of CAM with the semantic alignment of vision-language models (VLMs). Specifically, we derive channel-level semantic representations using CLIP embeddings and linear discriminant analysis, and aggregate them with CAM weights to produce textual descriptions of salient visual evidence. This yields explanations that jointly specify where the model attends and what visual attributes likely support its decision. We further extend TextCAM to generate feature channels into semantically coherent groups, enabling more fine-grained visual-textual explanations. Experiments on ImageNet, CLEVR, and CUB demonstrate that TextCAM produces faithful and interpretable rationales that improve human understanding, detect spurious correlations, and preserve model fidelity.