Optimizing against Infeasible Inclusions from Data for Semantic Segmentation through Morphology
This addresses the issue of absurd segmentations in semantic segmentation for computer vision applications, though it is incremental as it builds on existing data-driven paradigms.
The paper tackles the problem of semantic segmentation models producing infeasible segmentations, such as labeling a segment as 'road' when it is included in a 'sky' segment, by introducing a method that extracts and enforces spatial class constraints during training. The result is consistent and significant performance improvements across ADE20K, Cityscapes, and ACDC datasets.
State-of-the-art semantic segmentation models are typically optimized in a data-driven fashion, minimizing solely per-pixel or per-segment classification objectives on their training data. This purely data-driven paradigm often leads to absurd segmentations, especially when the domain of input images is shifted from the one encountered during training. For instance, state-of-the-art models may assign the label "road" to a segment that is included by another segment that is respectively labeled as "sky". However, the ground truth of the existing dataset at hand dictates that such inclusion is not feasible. Our method, Infeasible Semantic Inclusions (InSeIn), first extracts explicit inclusion constraints that govern spatial class relations from the semantic segmentation training set at hand in an offline, data-driven fashion, and then enforces a morphological yet differentiable loss that penalizes violations of these constraints during training to promote prediction feasibility. InSeIn is a light-weight plug-and-play method, constitutes a novel step towards minimizing infeasible semantic inclusions in the predictions of learned segmentation models, and yields consistent and significant performance improvements over diverse state-of-the-art networks across the ADE20K, Cityscapes, and ACDC datasets. https://github.com/SHAMIK-97/InSeIn