Evaluation Framework of Superpixel Methods with a Global Regularity Measure
This work addresses the need for more robust evaluation in computer vision for researchers and practitioners, though it is incremental as it builds on existing metrics and methods.
The paper tackles the problem of biased comparisons in superpixel methods by introducing an evaluation framework that unifies the process and addresses limitations in existing metrics, proposing a new global regularity measure (GR) that reduces bias and is correlated with application performances.
In the superpixel literature, the comparison of state-of-the-art methods can be biased by the non-robustness of some metrics to decomposition aspects, such as the superpixel scale. Moreover, most recent decomposition methods allow to set a shape regularity parameter, which can have a substantial impact on the measured performances. In this paper, we introduce an evaluation framework, that aims to unify the comparison process of superpixel methods. We investigate the limitations of existing metrics, and propose to evaluate each of the three core decomposition aspects: color homogeneity, respect of image objects and shape regularity. To measure the regularity aspect, we propose a new global regularity measure (GR), which addresses the non-robustness of state-of-the-art metrics. We evaluate recent superpixel methods with these criteria, at several superpixel scales and regularity levels. The proposed framework reduces the bias in the comparison process of state-of-the-art superpixel methods. Finally, we demonstrate that the proposed GR measure is correlated with the performances of various applications.