On the Explainability of Vision-Language Models in Art History
This work addresses the need for interpretability in VLMs for art history experts, but it is incremental as it applies existing XAI methods to a new domain without major methodological breakthroughs.
The paper tackled the problem of making vision-language models (VLMs) like CLIP explainable in art history by evaluating seven XAI methods, finding that their effectiveness depends on the conceptual stability and representational availability of categories, with results based on zero-shot localization and human studies.
Vision-Language Models (VLMs) transfer visual and textual data into a shared embedding space. In so doing, they enable a wide range of multimodal tasks, while also raising critical questions about the nature of machine 'understanding.' In this paper, we examine how Explainable Artificial Intelligence (XAI) methods can render the visual reasoning of a VLM - namely, CLIP - legible in art-historical contexts. To this end, we evaluate seven methods, combining zero-shot localization experiments with human interpretability studies. Our results indicate that, while these methods capture some aspects of human interpretation, their effectiveness hinges on the conceptual stability and representational availability of the examined categories.