Evaluating Large-Vocabulary Object Detectors: The Devil is in the Details
This addresses a critical evaluation issue for researchers and practitioners in computer vision, particularly for large-vocabulary object detection, by providing more reliable metrics, though it is incremental as it builds on existing AP frameworks.
The paper tackles the problem that the standard average precision (AP) metric for object detection is not category-independent and can be gamed, especially in large-vocabulary settings like LVIS, by showing a re-ranking policy can improve AP by a large margin; it introduces two new metrics, including a fix to AP and AP-Pool, and finds that explicit calibration improves state-of-the-art on AP-Pool by 1.7 points.
By design, average precision (AP) for object detection aims to treat all classes independently: AP is computed independently per category and averaged. On one hand, this is desirable as it treats all classes equally. On the other hand, it ignores cross-category confidence calibration, a key property in real-world use cases. Unfortunately, under important conditions (i.e., large vocabulary, high instance counts) the default implementation of AP is neither category independent, nor does it directly reward properly calibrated detectors. In fact, we show that on LVIS the default implementation produces a gameable metric, where a simple, un-intuitive re-ranking policy can improve AP by a large margin. To address these limitations, we introduce two complementary metrics. First, we present a simple fix to the default AP implementation, ensuring that it is independent across categories as originally intended. We benchmark recent LVIS detection advances and find that many reported gains do not translate to improvements under our new evaluation, suggesting recent improvements may arise from difficult to interpret changes to cross-category rankings. Given the importance of reliably benchmarking cross-category rankings, we consider a pooled version of AP (AP-Pool) that rewards properly calibrated detectors by directly comparing cross-category rankings. Finally, we revisit classical approaches for calibration and find that explicitly calibrating detectors improves state-of-the-art on AP-Pool by 1.7 points