VLLaVO: Mitigating Visual Gap through LLMs
This addresses domain shift issues in real-world visual applications, but it is incremental as it builds on existing cross-domain learning by incorporating text modality.
The paper tackles the problem of domain shift in visual cross-domain learning by proposing VLLaVO, which combines vision-language and large language models to convert images into textual descriptions and finetune on them, achieving effectiveness in domain generalization and unsupervised domain adaptation settings.
Recent advances achieved by deep learning models rely on the independent and identically distributed assumption, hindering their applications in real-world scenarios with domain shifts. To tackle this issue, cross-domain learning aims at extracting domain-invariant knowledge to reduce the domain shift between training and testing data. However, in visual cross-domain learning, traditional methods concentrate solely on the image modality, disregarding the potential benefits of incorporating the text modality. In this work, we propose VLLaVO, combining Vision language models and Large Language models as Visual cross-dOmain learners. VLLaVO uses vision-language models to convert images into detailed textual descriptions. A large language model is then finetuned on textual descriptions of the source/target domain generated by a designed instruction template. Extensive experimental results under domain generalization and unsupervised domain adaptation settings demonstrate the effectiveness of the proposed method.