Cross-Domain Semantic Segmentation on Inconsistent Taxonomy using VLMs
This addresses a key limitation in domain adaptation for semantic segmentation, enabling recognition of new classes in target domains without labeled data, though it builds incrementally on existing UDA methods.
The paper tackles semantic segmentation in unsupervised domain adaptation with mismatched class taxonomies between source and target domains, introducing CSI, which uses vision language models to relabel new classes without target ground truth, achieving effective adaptation across benchmarks like coarse-to-fine and open taxonomy settings.
The challenge of semantic segmentation in Unsupervised Domain Adaptation (UDA) emerges not only from domain shifts between source and target images but also from discrepancies in class taxonomies across domains. Traditional UDA research assumes consistent taxonomy between the source and target domains, thereby limiting their ability to recognize and adapt to the taxonomy of the target domain. This paper introduces a novel approach, Cross-Domain Semantic Segmentation on Inconsistent Taxonomy using Vision Language Models (CSI), which effectively performs domain-adaptive semantic segmentation even in situations of source-target class mismatches. CSI leverages the semantic generalization potential of Visual Language Models (VLMs) to create synergy with previous UDA methods. It leverages segment reasoning obtained through traditional UDA methods, combined with the rich semantic knowledge embedded in VLMs, to relabel new classes in the target domain. This approach allows for effective adaptation to extended taxonomies without requiring any ground truth label for the target domain. Our method has shown to be effective across various benchmarks in situations of inconsistent taxonomy settings (coarse-to-fine taxonomy and open taxonomy) and demonstrates consistent synergy effects when integrated with previous state-of-the-art UDA methods. The implementation is available at http://github.com/jkee58/CSI.