Revisiting Knowledge Distillation for Object Detection
This work addresses the problem of reducing labeled data dependency in object detection distillation for computer vision researchers, offering incremental improvements by relaxing constraints and enabling new properties like using unlabeled data.
The paper tackles the constraint in object detection distillation that requires both a teacher model and ground-truth labels by proposing a framework where a student is trained with pseudo labels from the teacher and then fine-tuned with labeled data, resulting in improved performance over existing algorithms and achieving the same performance with only 20% of COCO labels as with the full set.
The existing solutions for object detection distillation rely on the availability of both a teacher model and ground-truth labels. We propose a new perspective to relax this constraint. In our framework, a student is first trained with pseudo labels generated by the teacher, and then fine-tuned using labeled data, if any available. Extensive experiments demonstrate improvements over existing object detection distillation algorithms. In addition, decoupling the teacher and ground-truth distillation in this framework provides interesting properties such: as 1) using unlabeled data to further improve the student's performance, 2) combining multiple teacher models of different architectures, even with different object categories, and 3) reducing the need for labeled data (with only 20% of COCO labels, this method achieves the same performance as the model trained on the entire set of labels). Furthermore, a by-product of this approach is the potential usage for domain adaptation. We verify these properties through extensive experiments.