Unlocking Robust Semantic Segmentation Performance via Label-only Elastic Deformations against Implicit Label Noise
This addresses implicit label noise in semantic segmentation, which is an incremental improvement over existing augmentation methods.
The paper tackles the problem of subtle label imperfections in semantic segmentation datasets by introducing NSegment+, a data augmentation framework that applies elastic deformations only to labels while keeping images unchanged. The method achieves mIoU gains of up to +2.29, +2.38, +1.75, and +3.39 on Vaihingen, LoveDA, Cityscapes, and PASCAL VOC datasets respectively.
While previous studies on image segmentation focus on handling severe (or explicit) label noise, real-world datasets also exhibit subtle (or implicit) label imperfections. These arise from inherent challenges, such as ambiguous object boundaries and annotator variability. Although not explicitly present, such mild and latent noise can still impair model performance. Typical data augmentation methods, which apply identical transformations to the image and its label, risk amplifying these subtle imperfections and limiting the model's generalization capacity. In this paper, we introduce NSegment+, a novel augmentation framework that decouples image and label transformations to address such realistic noise for semantic segmentation. By introducing controlled elastic deformations only to segmentation labels while preserving the original images, our method encourages models to focus on learning robust representations of object structures despite minor label inconsistencies. Extensive experiments demonstrate that NSegment+ consistently improves performance, achieving mIoU gains of up to +2.29, +2.38, +1.75, and +3.39 in average on Vaihingen, LoveDA, Cityscapes, and PASCAL VOC, respectively-even without bells and whistles, highlighting the importance of addressing implicit label noise. These gains can be further amplified when combined with other training tricks, including CutMix and Label Smoothing.