Reliability in Semantic Segmentation: Are We on the Right Track?
This work addresses the critical need for reliable semantic segmentation models in real-world applications, providing a comprehensive analysis that is novel for this domain but incremental in its methodological approach.
The study evaluates the reliability of modern semantic segmentation models, including transformers and ResNet-based architectures, across robustness, calibration, misclassification detection, and out-of-distribution detection, finding that recent models are more robust but not more reliable in uncertainty estimation, and that improving calibration can enhance other uncertainty metrics.
Motivated by the increasing popularity of transformers in computer vision, in recent times there has been a rapid development of novel architectures. While in-domain performance follows a constant, upward trend, properties like robustness or uncertainty estimation are less explored -leaving doubts about advances in model reliability. Studies along these axes exist, but they are mainly limited to classification models. In contrast, we carry out a study on semantic segmentation, a relevant task for many real-world applications where model reliability is paramount. We analyze a broad variety of models, spanning from older ResNet-based architectures to novel transformers and assess their reliability based on four metrics: robustness, calibration, misclassification detection and out-of-distribution (OOD) detection. We find that while recent models are significantly more robust, they are not overall more reliable in terms of uncertainty estimation. We further explore methods that can come to the rescue and show that improving calibration can also help with other uncertainty metrics such as misclassification or OOD detection. This is the first study on modern segmentation models focused on both robustness and uncertainty estimation and we hope it will help practitioners and researchers interested in this fundamental vision task. Code available at https://github.com/naver/relis.