YOLOBench: Benchmarking Efficient Object Detectors on Embedded Systems
This work provides a comprehensive benchmarking tool for researchers and practitioners in embedded systems to compare efficient object detectors, though it is incremental as it builds on existing YOLO models and evaluation methods.
The authors introduced YOLOBench, a benchmark for evaluating 550+ YOLO-based object detection models across multiple datasets and embedded hardware platforms, revealing that older models like YOLOv3 and YOLOv4 can achieve competitive accuracy-latency trade-offs when modern techniques are applied, and demonstrated that some zero-cost accuracy estimators can effectively identify Pareto-optimal models, such as one competitive with YOLOv8 on a Raspberry Pi 4 CPU.
We present YOLOBench, a benchmark comprised of 550+ YOLO-based object detection models on 4 different datasets and 4 different embedded hardware platforms (x86 CPU, ARM CPU, Nvidia GPU, NPU). We collect accuracy and latency numbers for a variety of YOLO-based one-stage detectors at different model scales by performing a fair, controlled comparison of these detectors with a fixed training environment (code and training hyperparameters). Pareto-optimality analysis of the collected data reveals that, if modern detection heads and training techniques are incorporated into the learning process, multiple architectures of the YOLO series achieve a good accuracy-latency trade-off, including older models like YOLOv3 and YOLOv4. We also evaluate training-free accuracy estimators used in neural architecture search on YOLOBench and demonstrate that, while most state-of-the-art zero-cost accuracy estimators are outperformed by a simple baseline like MAC count, some of them can be effectively used to predict Pareto-optimal detection models. We showcase that by using a zero-cost proxy to identify a YOLO architecture competitive against a state-of-the-art YOLOv8 model on a Raspberry Pi 4 CPU. The code and data are available at https://github.com/Deeplite/deeplite-torch-zoo