A Study on Unsupervised Domain Adaptation for Semantic Segmentation in the Era of Vision-Language Models
This work addresses domain adaptation challenges for autonomous driving systems, but it is incremental as it builds on existing methods by swapping encoders.
The study tackled the problem of domain shifts in semantic segmentation for autonomous driving by replacing the encoder in existing unsupervised domain adaptation methods with a vision-language pre-trained encoder, resulting in performance improvements of up to 10.0% mIoU on a synthetic-to-real domain shift and up to 13.7% mIoU on unseen domains.
Despite the recent progress in deep learning based computer vision, domain shifts are still one of the major challenges. Semantic segmentation for autonomous driving faces a wide range of domain shifts, e.g. caused by changing weather conditions, new geolocations and the frequent use of synthetic data in model training. Unsupervised domain adaptation (UDA) methods have emerged which adapt a model to a new target domain by only using unlabeled data of that domain. The variety of UDA methods is large but all of them use ImageNet pre-trained models. Recently, vision-language models have demonstrated strong generalization capabilities which may facilitate domain adaptation. We show that simply replacing the encoder of existing UDA methods like DACS by a vision-language pre-trained encoder can result in significant performance improvements of up to 10.0% mIoU on the GTA5-to-Cityscapes domain shift. For the generalization performance to unseen domains, the newly employed vision-language pre-trained encoder provides a gain of up to 13.7% mIoU across three unseen datasets. However, we find that not all UDA methods can be easily paired with the new encoder and that the UDA performance does not always likewise transfer into generalization performance. Finally, we perform our experiments on an adverse weather condition domain shift to further verify our findings on a pure real-to-real domain shift.