RT-DETRv4: Painlessly Furthering Real-Time Object Detection with Vision Foundation Models
This work addresses performance limitations in real-time object detection for on-device deployment, representing an incremental improvement through a novel distillation approach.
The paper tackles the problem of degraded feature representation in lightweight real-time object detectors by proposing a distillation framework that leverages Vision Foundation Models (VFMs) to enhance performance without increasing inference overhead. The resulting RT-DETRv4 model achieves state-of-the-art results on COCO with AP scores up to 57.0 at 78 FPS.
Real-time object detection has achieved substantial progress through meticulously designed architectures and optimization strategies. However, the pursuit of high-speed inference via lightweight network designs often leads to degraded feature representation, which hinders further performance improvements and practical on-device deployment. In this paper, we propose a cost-effective and highly adaptable distillation framework that harnesses the rapidly evolving capabilities of Vision Foundation Models (VFMs) to enhance lightweight object detectors. Given the significant architectural and learning objective disparities between VFMs and resource-constrained detectors, achieving stable and task-aligned semantic transfer is challenging. To address this, on one hand, we introduce a Deep Semantic Injector (DSI) module that facilitates the integration of high-level representations from VFMs into the deep layers of the detector. On the other hand, we devise a Gradient-guided Adaptive Modulation (GAM) strategy, which dynamically adjusts the intensity of semantic transfer based on gradient norm ratios. Without increasing deployment and inference overhead, our approach painlessly delivers striking and consistent performance gains across diverse DETR-based models, underscoring its practical utility for real-time detection. Our new model family, RT-DETRv4, achieves state-of-the-art results on COCO, attaining AP scores of 49.7/53.5/55.4/57.0 at corresponding speeds of 273/169/124/78 FPS.