RbA: Segmenting Unknown Regions Rejected by All
This work addresses the challenge of identifying novel objects in segmentation for applications like autonomous driving, offering a method that avoids performance degradation on known classes, though it is incremental in building upon existing mask classification techniques.
The paper tackles the problem of segmenting unknown objects in semantic segmentation by proposing a region-level classification paradigm and a novel outlier scoring function called RbA, which defines outliers as regions rejected by all known classes, achieving improved performance without degrading known class accuracy.
Standard semantic segmentation models owe their success to curated datasets with a fixed set of semantic categories, without contemplating the possibility of identifying unknown objects from novel categories. Existing methods in outlier detection suffer from a lack of smoothness and objectness in their predictions, due to limitations of the per-pixel classification paradigm. Furthermore, additional training for detecting outliers harms the performance of known classes. In this paper, we explore another paradigm with region-level classification to better segment unknown objects. We show that the object queries in mask classification tend to behave like one \vs all classifiers. Based on this finding, we propose a novel outlier scoring function called RbA by defining the event of being an outlier as being rejected by all known classes. Our extensive experiments show that mask classification improves the performance of the existing outlier detection methods, and the best results are achieved with the proposed RbA. We also propose an objective to optimize RbA using minimal outlier supervision. Further fine-tuning with outliers improves the unknown performance, and unlike previous methods, it does not degrade the inlier performance.