MeshMetrics: A Precise Implementation of Distance-Based Image Segmentation Metrics
This solves the problem of inconsistent metric evaluations for researchers in image segmentation, though it is incremental as it focuses on implementation improvements rather than new metrics.
The paper tackles the reproducibility crisis in image segmentation by addressing unreliable implementations of distance-based metrics, introducing MeshMetrics, a mesh-based framework that achieves higher accuracy and precision than existing tools, with discrepancies reduced from over 100 mm for Hausdorff distance and 30%pt for normalized surface distance.
The surge of research in image segmentation has yielded remarkable performance gains but also exposed a reproducibility crisis. A major contributor is performance evaluation, where both selection and implementation of metrics play critical roles. While recent efforts have improved the former, the reliability of metric implementation has received far less attention. Pitfalls in distance-based metric implementation can lead to considerable discrepancies between common open-source tools, for instance, exceeding 100 mm for the Hausdorff distance and 30%pt for the normalized surface distance for the same pair of segmentations. To address these pitfalls, we introduce MeshMetrics, a mesh-based framework that provides a more precise computation of distance-based metrics than conventional grid-based approaches. Through theoretical analysis and empirical validation, we demonstrate that MeshMetrics achieves higher accuracy and precision than established tools, and is substantially less affected by discretization artifacts, such as distance quantization. We release MeshMetrics as an open-source Python package, available at https://github.com/gasperpodobnik/MeshMetrics.