Interpreting ResNet-based CLIP via Neuron-Attention Decomposition
This work addresses interpretability in vision-language models for researchers and practitioners, but it is incremental as it builds on existing CLIP architectures.
The authors tackled the problem of interpreting neurons in CLIP-ResNet by decomposing their contributions into neuron-head pairs, finding that these pairs can be approximated by a single direction in the embedding space and are useful for applications like semantic segmentation and monitoring dataset shifts, with results showing outperformance in segmentation tasks.
We present a novel technique for interpreting the neurons in CLIP-ResNet by decomposing their contributions to the output into individual computation paths. More specifically, we analyze all pairwise combinations of neurons and the following attention heads of CLIP's attention-pooling layer. We find that these neuron-head pairs can be approximated by a single direction in CLIP-ResNet's image-text embedding space. Leveraging this insight, we interpret each neuron-head pair by associating it with text. Additionally, we find that only a sparse set of the neuron-head pairs have a significant contribution to the output value, and that some neuron-head pairs, while polysemantic, represent sub-concepts of their corresponding neurons. We use these observations for two applications. First, we employ the pairs for training-free semantic segmentation, outperforming previous methods for CLIP-ResNet. Second, we utilize the contributions of neuron-head pairs to monitor dataset distribution shifts. Our results demonstrate that examining individual computation paths in neural networks uncovers interpretable units, and that such units can be utilized for downstream tasks.