Generate, Transduct, Adapt: Iterative Transduction with VLMs
This work addresses a gap in leveraging language space structure for transductive zero-shot learning, offering incremental improvements for vision-language model applications.
The paper tackles the problem of transductive zero-shot learning with vision-language models by proposing GTA-CLIP, which incorporates language model supervision for joint transduction in language and vision spaces, resulting in an average performance improvement of 8.6% over CLIP and 3.7% over transductive CLIP across 12 datasets and 3 encoders.
Transductive zero-shot learning with vision-language models leverages image-image similarities within the dataset to achieve better classification accuracy compared to the inductive setting. However, there is little work that explores the structure of the language space in this context. We propose GTA-CLIP, a novel technique that incorporates supervision from language models for joint transduction in language and vision spaces. Our approach is iterative and consists of three steps: (i) incrementally exploring the attribute space by querying language models, (ii) an attribute-augmented transductive inference procedure, and (iii) fine-tuning the language and vision encoders based on inferred labels within the dataset. Through experiments with CLIP encoders, we demonstrate that GTA-CLIP, yields an average performance improvement of 8.6% and 3.7% across 12 datasets and 3 encoders, over CLIP and transductive CLIP respectively in the zero-shot setting. We also observe similar improvements in a few-shot setting. We present ablation studies that demonstrate the value of each step and visualize how the vision and language spaces evolve over iterations driven by the transductive learning. Code is released at https://github.com/cvl-umass/GTA-CLIP