Benchmarking Segmentation Models with Mask-Preserved Attribute Editing
This work addresses the need for more comprehensive robustness evaluation in segmentation models, particularly for practitioners deploying them in varied scenes, though it is incremental as it builds on existing evaluation paradigms by adding local attribute considerations.
The paper tackles the problem of evaluating segmentation models' robustness by investigating both local and global attribute variations, constructing a mask-preserved attribute editing pipeline to edit real images while preserving structural information for label reuse. It finds that both types of variations affect performance, with models showing divergent sensitivities across variation types, and argues for including local attributes in robustness benchmarks.
When deploying segmentation models in practice, it is critical to evaluate their behaviors in varied and complex scenes. Different from the previous evaluation paradigms only in consideration of global attribute variations (e.g. adverse weather), we investigate both local and global attribute variations for robustness evaluation. To achieve this, we construct a mask-preserved attribute editing pipeline to edit visual attributes of real images with precise control of structural information. Therefore, the original segmentation labels can be reused for the edited images. Using our pipeline, we construct a benchmark covering both object and image attributes (e.g. color, material, pattern, style). We evaluate a broad variety of semantic segmentation models, spanning from conventional close-set models to recent open-vocabulary large models on their robustness to different types of variations. We find that both local and global attribute variations affect segmentation performances, and the sensitivity of models diverges across different variation types. We argue that local attributes have the same importance as global attributes, and should be considered in the robustness evaluation of segmentation models. Code: https://github.com/PRIS-CV/Pascal-EA.