Interpreting and Analysing CLIP's Zero-Shot Image Classification via Mutual Knowledge
This provides a human-friendly way to understand zero-shot classification decisions for researchers and practitioners using CLIP, but it is incremental as it builds on existing CLIP models without major breakthroughs.
The paper tackled the problem of interpreting CLIP's zero-shot image classification by analyzing mutual knowledge between vision and language encoders, resulting in a textual concept-based explanation method that effectively interprets classification decisions across 13 CLIP models.
Contrastive Language-Image Pretraining (CLIP) performs zero-shot image classification by mapping images and textual class representation into a shared embedding space, then retrieving the class closest to the image. This work provides a new approach for interpreting CLIP models for image classification from the lens of mutual knowledge between the two modalities. Specifically, we ask: what concepts do both vision and language CLIP encoders learn in common that influence the joint embedding space, causing points to be closer or further apart? We answer this question via an approach of textual concept-based explanations, showing their effectiveness, and perform an analysis encompassing a pool of 13 CLIP models varying in architecture, size and pretraining datasets. We explore those different aspects in relation to mutual knowledge, and analyze zero-shot predictions. Our approach demonstrates an effective and human-friendly way of understanding zero-shot classification decisions with CLIP.