Generalize or Detect? Towards Robust Semantic Segmentation Under Multiple Distribution Shifts
This addresses the challenge of robust segmentation in open-world scenarios for safety-critical applications like autonomous driving, representing a novel integration rather than an incremental improvement.
The paper tackles the problem of semantic segmentation under multiple distribution shifts by enabling models to generalize to covariate shifts while detecting semantic shifts, achieving state-of-the-art performance in both OOD detection and domain generalization across benchmarks.
In open-world scenarios, where both novel classes and domains may exist, an ideal segmentation model should detect anomaly classes for safety and generalize to new domains. However, existing methods often struggle to distinguish between domain-level and semantic-level distribution shifts, leading to poor out-of-distribution (OOD) detection or domain generalization performance. In this work, we aim to equip the model to generalize effectively to covariate-shift regions while precisely identifying semantic-shift regions. To achieve this, we design a novel generative augmentation method to produce coherent images that incorporate both anomaly (or novel) objects and various covariate shifts at both image and object levels. Furthermore, we introduce a training strategy that recalibrates uncertainty specifically for semantic shifts and enhances the feature extractor to align features associated with domain shifts. We validate the effectiveness of our method across benchmarks featuring both semantic and domain shifts. Our method achieves state-of-the-art performance across all benchmarks for both OOD detection and domain generalization. Code is available at https://github.com/gaozhitong/MultiShiftSeg.