Language-assisted Vision Model Debugger: A Sample-Free Approach to Finding and Fixing Bugs
This work addresses the challenge of debugging vision models for researchers and practitioners by reducing annotation effort, though it is incremental as it builds on existing multi-modal models.
The authors tackled the problem of diagnosing systematic errors in vision models without requiring annotated image samples by proposing a language-assisted method using CLIP and LLMs to find bugs via text descriptions. They validated their approach on Waterbirds and CelebA datasets, successfully identifying both known and previously unknown bugs.
Vision models with high overall accuracy often exhibit systematic errors in specific scenarios, posing potential serious safety concerns. Diagnosing bugs of vision models is gaining increased attention, however traditional diagnostic approaches require annotation efforts (eg rich metadata accompanying each samples of CelebA). To address this issue,We propose a language-assisted diagnostic method that uses texts instead of images to diagnose bugs in vision models based on multi-modal models (eg CLIP). Our approach connects the embedding space of CLIP with the buggy vision model to be diagnosed; meanwhile, utilizing a shared classifier and the cross-modal transferability of embedding space from CLIP, the text-branch of CLIP become a proxy model to find bugs in the buggy model. The proxy model can classify texts paired with images. During the diagnosis, a Large Language Model (LLM) is employed to obtain task-relevant corpora, and this corpora is used to extract keywords. Descriptions constructed with templates containing these keywords serve as input text to probe errors in the proxy model. Finally, we validate the ability to diagnose existing visual models using language on the Waterbirds and CelebA datasets, we can identify bugs comprehensible to human experts, uncovering not only known bugs but also previously unknown ones.