Towards Integrating Uncertainty for Domain-Agnostic Segmentation
This work addresses the challenge of robust segmentation for users in varied domains, but it is incremental as it builds on existing models with preliminary refinement benefits.
The paper tackled the problem of foundation models for segmentation being vulnerable in shifted or limited-knowledge domains by investigating if uncertainty quantification can enhance generalisability, resulting in a benchmark (UncertSAM) and findings that a last-layer Laplace approximation yields uncertainty estimates correlating well with segmentation errors.
Foundation models for segmentation such as the Segment Anything Model (SAM) family exhibit strong zero-shot performance, but remain vulnerable in shifted or limited-knowledge domains. This work investigates whether uncertainty quantification can mitigate such challenges and enhance model generalisability in a domain-agnostic manner. To this end, we (1) curate UncertSAM, a benchmark comprising eight datasets designed to stress-test SAM under challenging segmentation conditions including shadows, transparency, and camouflage; (2) evaluate a suite of lightweight, post-hoc uncertainty estimation methods; and (3) assess a preliminary uncertainty-guided prediction refinement step. Among evaluated approaches, a last-layer Laplace approximation yields uncertainty estimates that correlate well with segmentation errors, indicating a meaningful signal. While refinement benefits are preliminary, our findings underscore the potential of incorporating uncertainty into segmentation models to support robust, domain-agnostic performance. Our benchmark and code are made publicly available.