Sensitivity of Average Precision to Bounding Box Perturbations
This work addresses a fundamental evaluation issue in computer vision, revealing why improving mAP becomes harder as models advance, which is incremental but important for researchers and practitioners.
The paper quantifies the sensitivity of Average Precision (AP) to bounding box perturbations in object detection, showing that small translations cause significant drops in mAP, such as an 8.4% decrease with a one-pixel shift.
Object detection is a fundamental vision task. It has been highly researched in academia and has been widely adopted in industry. Average Precision (AP) is the standard score for evaluating object detectors. Our understanding of the subtleties of this score, however, is limited. Here, we quantify the sensitivity of AP to bounding box perturbations and show that AP is very sensitive to small translations. Only one pixel shift is enough to drop the mAP of a model by 8.4%. The mAP drop over small objects with only one pixel shift is 23.1%. The corresponding numbers when ground-truth (GT) boxes are used as predictions are 23% and 41.7%, respectively. These results explain why achieving higher mAP becomes increasingly harder as models get better. We also investigate the effect of box scaling on AP. Code and data is available at https://github.com/aliborji/AP_Box_Perturbation.