DETRDistill: A Universal Knowledge Distillation Framework for DETR-families
This addresses the deployment challenge of DETR-based detectors for applications requiring efficient object detection, though it is incremental as it adapts existing distillation techniques to a specific model family.
The paper tackles the problem of compressing large transformer-based object detectors (DETRs) for real-world deployment by proposing DETRDistill, a knowledge distillation framework that improves various DETR models by over 2.0 mAP on the COCO dataset, even surpassing teacher models.
Transformer-based detectors (DETRs) are becoming popular for their simple framework, but the large model size and heavy time consumption hinder their deployment in the real world. While knowledge distillation (KD) can be an appealing technique to compress giant detectors into small ones for comparable detection performance and low inference cost. Since DETRs formulate object detection as a set prediction problem, existing KD methods designed for classic convolution-based detectors may not be directly applicable. In this paper, we propose DETRDistill, a novel knowledge distillation method dedicated to DETR-families. Specifically, we first design a Hungarian-matching logits distillation to encourage the student model to have the exact predictions as that of teacher DETRs. Next, we propose a target-aware feature distillation to help the student model learn from the object-centric features of the teacher model. Finally, in order to improve the convergence rate of the student DETR, we introduce a query-prior assignment distillation to speed up the student model learning from well-trained queries and stable assignment of the teacher model. Extensive experimental results on the COCO dataset validate the effectiveness of our approach. Notably, DETRDistill consistently improves various DETRs by more than 2.0 mAP, even surpassing their teacher models.