USB: Universal-Scale Object Detection Benchmark
This work tackles the problem of limited scale variation and unfair benchmarking in object detection for researchers and practitioners, though it is incremental as it builds upon existing datasets and methods.
The paper introduces the Universal-Scale object detection Benchmark (USB) to address insufficient scale variation and unfair comparison protocols in existing benchmarks like COCO, by incorporating datasets such as Waymo Open and Manga109-s, and proposes training and evaluation protocols for fairer assessments, revealing weaknesses in COCO-biased methods through experiments with 15 methods.
Benchmarks, such as COCO, play a crucial role in object detection. However, existing benchmarks are insufficient in scale variation, and their protocols are inadequate for fair comparison. In this paper, we introduce the Universal-Scale object detection Benchmark (USB). USB has variations in object scales and image domains by incorporating COCO with the recently proposed Waymo Open Dataset and Manga109-s dataset. To enable fair comparison and inclusive research, we propose training and evaluation protocols. They have multiple divisions for training epochs and evaluation image resolutions, like weight classes in sports, and compatibility across training protocols, like the backward compatibility of the Universal Serial Bus. Specifically, we request participants to report results with not only higher protocols (longer training) but also lower protocols (shorter training). Using the proposed benchmark and protocols, we conducted extensive experiments using 15 methods and found weaknesses of existing COCO-biased methods. The code is available at https://github.com/shinya7y/UniverseNet .