Zero-Shot Textual Explanations via Translating Decision-Critical Features
This work addresses the need for transparent AI in image classification by providing zero-shot textual explanations, though it is incremental as it builds on existing vision-language models and feature alignment techniques.
The paper tackles the problem of generating textual explanations for image classifier decisions by proposing TEXTER, a method that isolates decision-critical features before alignment, resulting in more faithful and interpretable explanations than existing zero-shot methods.
Textual explanations make image classifier decisions transparent by describing the prediction rationale in natural language. Large vision-language models can generate captions but are designed for general visual understanding, not classifier-specific reasoning. Existing zero-shot explanation methods align global image features with language, producing descriptions of what is visible rather than what drives the prediction. We propose TEXTER, which overcomes this limitation by isolating decision-critical features before alignment. TEXTER identifies the neurons contributing to the prediction and emphasizes the features encoded in those neurons -- i.e., the decision-critical features. It then maps these emphasized features into the CLIP feature space to retrieve textual explanations that reflect the model's reasoning. A sparse autoencoder further improves interpretability, particularly for Transformer architectures. Extensive experiments show that TEXTER generates more faithful and interpretable explanations than existing methods. The code will be publicly released.