Domain Adaptive Video Semantic Segmentation via Cross-Domain Moving Object Mixing
This work addresses domain adaptation for video semantic segmentation, which is important for applications like autonomous driving, but it appears incremental as it builds on existing domain adaptation techniques with video-specific modifications.
The paper tackles the problem of domain bias in video semantic segmentation by proposing Cross-domain Moving Object Mixing (CMOM) and Feature Alignment with Temporal Context (FATC), achieving mIoU scores of 53.81% on VIPER to Cityscapes-Seq and 56.31% on SYNTHIA-Seq to Cityscapes-Seq, surpassing state-of-the-art methods.
The network trained for domain adaptation is prone to bias toward the easy-to-transfer classes. Since the ground truth label on the target domain is unavailable during training, the bias problem leads to skewed predictions, forgetting to predict hard-to-transfer classes. To address this problem, we propose Cross-domain Moving Object Mixing (CMOM) that cuts several objects, including hard-to-transfer classes, in the source domain video clip and pastes them into the target domain video clip. Unlike image-level domain adaptation, the temporal context should be maintained to mix moving objects in two different videos. Therefore, we design CMOM to mix with consecutive video frames, so that unrealistic movements are not occurring. We additionally propose Feature Alignment with Temporal Context (FATC) to enhance target domain feature discriminability. FATC exploits the robust source domain features, which are trained with ground truth labels, to learn discriminative target domain features in an unsupervised manner by filtering unreliable predictions with temporal consensus. We demonstrate the effectiveness of the proposed approaches through extensive experiments. In particular, our model reaches mIoU of 53.81% on VIPER to Cityscapes-Seq benchmark and mIoU of 56.31% on SYNTHIA-Seq to Cityscapes-Seq benchmark, surpassing the state-of-the-art methods by large margins. The code is available at: https://github.com/kyusik-cho/CMOM.