Handling new target classes in semantic segmentation with domain adaptation
This addresses a novel domain adaptation challenge in semantic segmentation for applications like autonomous driving, where new object classes may appear in different environments, though it is incremental in combining existing techniques.
The paper tackles the problem of semantic segmentation with domain adaptation when the target domain includes novel classes not present in the source, aiming for explicit prediction of these new classes. It proposes a framework combining domain adaptation and zero-shot learning, achieving significant performance gains over baselines across multiple benchmarks.
In this work, we define and address a novel domain adaptation (DA) problem in semantic scene segmentation, where the target domain not only exhibits a data distribution shift w.r.t. the source domain, but also includes novel classes that do not exist in the latter. Different to "open-set" and "universal domain adaptation", which both regard all objects from new classes as "unknown", we aim at explicit test-time prediction for these new classes. To reach this goal, we propose a framework that leverages domain adaptation and zero-shot learning techniques to enable "boundless" adaptation in the target domain. It relies on a novel architecture, along with a dedicated learning scheme, to bridge the source-target domain gap while learning how to map new classes' labels to relevant visual representations. The performance is further improved using self-training on target-domain pseudo-labels. For validation, we consider different domain adaptation set-ups, namely synthetic-2-real, country-2-country and dataset-2-dataset. Our framework outperforms the baselines by significant margins, setting competitive standards on all benchmarks for the new task. Code and models are available at https://github.com/valeoai/buda.