From Weights to Concepts: Data-Free Interpretability of CLIP via Singular Vector Decomposition
This addresses the need for interpretability in deployed vision-language models, offering a dataset-independent method that is incremental over existing activation-based approaches.
The paper tackled the problem of understanding vision-language models like CLIP by introducing SITH, a data-free framework that analyzes attention heads in weight space using singular vector decomposition and a new algorithm for interpretability, resulting in coherent explanations and enabling model edits that improve downstream performance without retraining.
As vision-language models are deployed at scale, understanding their internal mechanisms becomes increasingly critical. Existing interpretability methods predominantly rely on activations, making them dataset-dependent, vulnerable to data bias, and often restricted to coarse head-level explanations. We introduce SITH (Semantic Inspection of Transformer Heads), a fully data-free, training-free framework that directly analyzes CLIP's vision transformer in weight space. For each attention head, we decompose its value-output matrix into singular vectors and interpret each one via COMP (Coherent Orthogonal Matching Pursuit), a new algorithm that explains them as sparse, semantically coherent combinations of human-interpretable concepts. We show that SITH yields coherent, faithful intra-head explanations, validated through reconstruction fidelity and interpretability experiments. This allows us to use SITH for precise, interpretable weight-space model edits that amplify or suppress specific concepts, improving downstream performance without retraining. Furthermore, we use SITH to study model adaptation, showing how fine-tuning primarily reweights a stable semantic basis rather than learning entirely new features.