Measuring the Accuracy of Object Detectors and Trackers
This addresses a gap in evaluation for computer vision researchers working with segmented data, offering a more accurate tool for assessing detection and tracking performance, though it is incremental as it builds on existing IoU concepts.
The paper tackles the problem of evaluating object detectors and trackers on densely segmented datasets, where traditional box-based measures like IoU are inadequate, by introducing the relative Intersection over Union (rIoU) measure, which normalizes IoU with the optimal box for segmentation to provide a more precise accuracy measure ranging from 0 to 1.
The accuracy of object detectors and trackers is most commonly evaluated by the Intersection over Union (IoU) criterion. To date, most approaches are restricted to axis-aligned or oriented boxes and, as a consequence, many datasets are only labeled with boxes. Nevertheless, axis-aligned or oriented boxes cannot accurately capture an object's shape. To address this, a number of densely segmented datasets has started to emerge in both the object detection and the object tracking communities. However, evaluating the accuracy of object detectors and trackers that are restricted to boxes on densely segmented data is not straightforward. To close this gap, we introduce the relative Intersection over Union (rIoU) accuracy measure. The measure normalizes the IoU with the optimal box for the segmentation to generate an accuracy measure that ranges between 0 and 1 and allows a more precise measurement of accuracies. Furthermore, it enables an efficient and easy way to understand scenes and the strengths and weaknesses of an object detection or tracking approach. We display how the new measure can be efficiently calculated and present an easy-to-use evaluation framework. The framework is tested on the DAVIS and the VOT2016 segmentations and has been made available to the community.