Exploiting Semantics in Adversarial Training for Image-Level Domain Adaptation
This work addresses the costly annotation issue for pixel-wise tasks like semantic segmentation by improving domain adaptation from synthetic to real data, though it is incremental as it builds on existing GAN-based translation methods.
The paper tackles the domain shift problem between synthetic and real images for semantic segmentation by introducing semantic constraints into a GAN-based image translation method, resulting in a 16% mIoU improvement over training on synthetic images alone.
Performance achievable by modern deep learning approaches are directly related to the amount of data used at training time. Unfortunately, the annotation process is notoriously tedious and expensive, especially for pixel-wise tasks like semantic segmentation. Recent works have proposed to rely on synthetically generated imagery to ease the training set creation. However, models trained on these kind of data usually under-perform on real images due to the well known issue of domain shift. We address this problem by learning a domain-to-domain image translation GAN to shrink the gap between real and synthetic images. Peculiarly to our method, we introduce semantic constraints into the generation process to both avoid artifacts and guide the synthesis. To prove the effectiveness of our proposal, we show how a semantic segmentation CNN trained on images from the synthetic GTA dataset adapted by our method can improve performance by more than 16% mIoU with respect to the same model trained on synthetic images.