Domain Adaptation with a Single Vision-Language Embedding
This addresses domain adaptation for computer vision applications where target data is scarce or unavailable, offering a novel approach that reduces data dependency.
The paper tackles the problem of domain adaptation requiring target data at training time by introducing a framework that uses a single vision-language embedding to mine multiple visual styles for feature augmentation, enabling zero-shot and one-shot unsupervised domain adaptation. The method outperforms baselines in semantic segmentation tasks.
Domain adaptation has been extensively investigated in computer vision but still requires access to target data at the training time, which might be difficult to obtain in some uncommon conditions. In this paper, we present a new framework for domain adaptation relying on a single Vision-Language (VL) latent embedding instead of full target data. First, leveraging a contrastive language-image pre-training model (CLIP), we propose prompt/photo-driven instance normalization (PIN). PIN is a feature augmentation method that mines multiple visual styles using a single target VL latent embedding, by optimizing affine transformations of low-level source features. The VL embedding can come from a language prompt describing the target domain, a partially optimized language prompt, or a single unlabeled target image. Second, we show that these mined styles (i.e., augmentations) can be used for zero-shot (i.e., target-free) and one-shot unsupervised domain adaptation. Experiments on semantic segmentation demonstrate the effectiveness of the proposed method, which outperforms relevant baselines in the zero-shot and one-shot settings.